Inspiration
Prototyping has revolutionized digital design—Figma made web and mobile seamless, Canva simplified graphic creation—but 3D and VR remain stuck in the past. Existing tools are slow, overly technical, and inaccessible to most creators. We wanted to change that by building Prototyp3D, the first effortless, AI-powered 3D/VR prototyping tool that generates real, working code and an easily interactive environment solely from natural language descriptions.
What it does
Prototyp3D enables anyone—developers, designers, educators, and entrepreneurs—to bring their 3D and VR ideas to life instantly. Users simply describe their vision in plain text, and our AI Agents go through an optimized pipeline to generate functional 3D/VR code, while rendering the results step by step for the user to interact with. It’s like having a virtual engineering team that plans, builds, tests, and refines your prototype—all without requiring coding skills.
How we built it
AI-Powered Prototyping Agent: Uses natural language processing to interpret user descriptions and generate structured development tasks. It breaks large projects into organized Jira-style tickets using OpenAI's GPT 4o model.
Autonomous Code Generation Pipeline: Uses OpenAI's GPT o3-mini-high model to generate/improve code given specific context and acceptance criteria (specified in tickets).
Iterative Debugging & Testing System: Feeds and runs fresh project code in multiple virtual machines; launches Scrapybara AI agents to interact and play with the running 3D environment (or provide error information) in order to evaluate and generate feedback based on given ticket goals. The AI agent “sees,” “clicks,” and “drags” elements in a virtual environment to verify functionality before iterating a debugging process or moving on to the next ticket after achieving a certain similarity score.
Intuitive Front End: A visually clean and modern interface designed as a one-stop shop for all development needs. It seamlessly integrates essential features like autosaving, on-demand compilation, full-screen mode, syntax highlighting, easy code copying, and quick downloads. Every element is optimized for a smooth and efficient workflow.
3D & VR Code Generation: Leverages frameworks like Three.js, Babylon.js, and WebXR to build real, interactive 3D experiences.
Challenges we ran into
- Balancing AI autonomy with user control: Ensuring users could guide and tweak the generated prototypes while keeping the process seamless.
- Debugging in 3D/VR environments: Creating a system that understands and interacts with complex 3D spaces like a human tester. We ensured that AI agents only used visual and interactive observations in order to feed unbiased feedback into our code generating pipeline.
Accomplishments that we're proud of
The scope and complexity of projects that Prototyp3d can create far outpaces anything current models can output. We are proud to have utilized AI agents to create a debugging/feedback pipeline that models the thought process of real seasoned engineers.
Developed a working end-to-end pipeline where users can describe ideas and receive functional 3D/VR applications in minutes.
We are proud to have created a customer oriented interface that allows users to not only see and edit the generated code but also interact with the rendered 3d environment itself.
What we learned
The amount of possibilities that AI agents unlock is incredible: the use of them to simulate human engineers unlocks the possibility of creating projects over 10 times as complex as a query to the latest models.
Natural language AI for software engineering is incredibly powerful, but structuring its output into reliable, functional code takes careful design . Speed and quality are hard to balance: we focused a lot on outputting quality code, but that meant sacrificing run times and having to iterate through long pipelines.
What's next for prototyp3d
Expanding VR Compatibility: Adding support for Unity, Unreal Engine, and other VR platforms. Interactive Queries: We want to flexibility to the next level, letting users specify wanted changes by interacting with the 3D environment itself. Collaboration Features: Enabling real-time, multiplayer prototyping so teams can build together.
An even bigger future: we think our ai agent pipeline can be generalized to more types of generation. It can be used to generate projects outside of the 3d space. It can be used to write essays. It can be used to generate videos. We think our pipeline unlocks massive potential for generating quality output, and think that this technology can be generalized and used for so many different products.
Log in or sign up for Devpost to join the conversation.