Inspiration
We were inspired by procedural generated games such as Minecraft, Terraria, and many countless others. As children, we were all mesmerised by these games that were seemingly endless and diverse. We identified these experiences as showing much potential for augmentation by generative AI – we wanted to participate in the exploration, hence our project idea was born.
What it does
Our project, Inner Worlds, is a place where you can submit a prompt and generative AI will create your own personal 3D environment. To make it even more personal, we allow users to walk through their generated world in 3rd person! With Inner Worlds, you can create beautiful sceneries of temples and parks, create a snowy Christmas town dwarfed by an enormous pyramid, or! make something crazy like a camping ground on an island with a volcano AND A PYRAMID!!! WHAT???? That’s crazy 😯
How we built it
Our game was created using the Unity game engine, using C# and ChatGPT API for dynamic, generative AI environment design in Unity. Each 3D environment we create draws from a Prefab library of over 200 items which generative AI mixes and matches to suit your individual preferences! Our game has a simple and intuitive UI and UX. On start, the user is prompted to enter a prompt that we will send to a ChatGPT model through its API using a UnityWebRequest. We have a pre-written system prompt with detailed instructions for the GPT. The GPT model outputs a structured JSON string with prefab names and positions, and the API response is parsed to extract this JSON. The EnvironmentLoader then uses the parsed json to instantiate the prefabs in the Unity scene. An innovative part of our solution is the use of Prefab 3D models. If we were to do text to 3D Generation with completely custom meshes, it would be very difficult and time consuming. Our prefabs approach makes it much faster and easier to generate a complete and visually pleasing 3D environment. An LLM like ChatGPT, especially better models like 4o, is also very good at understanding natural language, enabling easy, intuitive and creative user prompts.
Challenges we ran into
As first-time hackathon participants, we found it difficult to balance feasibility, creativity and innovation. It took us almost 6-7 hours to think about the project we wanted to work towards! Additionally, we had a lot of trouble with Git and it was our first time working collaboratively in code -- Initially, before we used Github, we were messaging each other on Discord with code so that each of us could come up with version updates! This was also many of our first times tackling Unity & Game Development, and we had a lot of trouble with developing code in this new environment.
Accomplishments that we're proud of
We were proud of being able to accomplish many firsts throughout this project! Throughout all the challenges, we were able to rise up to each of them, and we gained a lot of experience in Unity, C#, Game Development, and Generative AI & Prompt Engineering working with the ChatGPT API. We hope to tackle more Unity and AI related projects in the future to further apply and enhance our skills.
What we learned
Overall, we learned more about Git, collaboration, Unity & Game Development, and Generative AI & working with LLM APIs and Prompt Engineering. We learned that hackathons require preparation and practice. We are extremely excited to continue our journey into this exciting world.
What's next for Inner Worlds
An extension to this project is creating a feedback loop, where the user is constantly able to feed improvements/suggestions to the AI to dynamically update, add, and remove generated objects to the environment until the user is satisfied with the generated environment to allow for increased user customability. Moving forward, we hope to extend this project to be a general framework for creating Unity Generative AI 3D environments, where we can support large libraries of Prefabs to generate many different types of unique environments or combine the Generative AI of objects to come up with unique shapes.
Log in or sign up for Devpost to join the conversation.