Inspiration
The growing demand for immersive, interactive digital experiences in sectors like gaming, e-commerce, and education inspired us to explore 3D modelling through Generative AI. Traditional 3D design tools are powerful but often require significant expertise and time investment. Our goal was to simplify this process by leveraging GenAI to make 3D content creation more accessible, intuitive, and efficient—even for non-technical users.
What it does
Our project uses GenAI to convert 2D inputs, natural language prompts, or skeletal shapes into fully formed 3D models. The system intelligently interprets context and style cues to generate detailed, textured 3D outputs in a fraction of the time traditional modelling takes. The generated models are exportable for use in AR/VR environments, animation pipelines, or gaming engines.
How we built it
We trained a deep learning pipeline combining a text-to-image diffusion model with a 3D shape synthesis engine. We experimented with tools like OpenAI’s Point-E, DreamFusion, and Blender APIs to streamline the generation-to-rendering process. Our backend was built in Python, leveraging PyTorch and Hugging Face libraries, while the front-end allowed for prompt-based interaction.
Challenges we ran into
- Aligning the generated 3D geometries with realistic proportions and textures was tricky.
- Ensuring model fidelity across varied lighting and angles required extensive fine-tuning.
- Integration between natural language processing and 3D rendering engines was technically complex.
- Data availability for training high-quality models was limited in certain domains.
Accomplishments that we're proud of
- Successfully generating usable, realistic 3D models from abstract prompts.
- Achieving a user-friendly workflow that reduced modelling time significantly.
- Building an MVP that can be extended into various applications—from product prototyping to educational tools.
- Receiving positive feedback from early testers who used it in AR/VR sandbox environments.
What we learned
- The immense potential of GenAI in reducing entry barriers for design and content creation.
- How to balance creativity and control in generative systems.
- The importance of UX/UI design in AI-driven tools for real-world adoption.
- Deeper understanding of multimodal AI models and their interoperability.
What's next for 3D modelling using GENAI
- Integrating real-time feedback to refine models during generation.
- Expanding prompt capabilities with audio and sketch inputs.
- Collaborating with 3D artists to co-develop hybrid workflows.
- Enhancing realism with procedural material generation and animation features.
- Exploring domain-specific applications like medical imaging, fashion, and interior design.
