Inspiration

3D modeling has always seemed like something that should be more accessible. It's exciting to build your own objects and environments, but traditional modeling software can have a pretty steep learning curve, especially if you're just getting started or want to try out a quick idea.

We wanted to make it easier to get from an idea to an actual 3D scene, without taking away the fun of making things yourself. We also wanted to go beyond static models and make it easy to bring those creations to life through animation.

What it does

Art Laboratory is a browser-based 3D modeling and animation tool that combines natural language with a hands-on editor.

You can describe what you want to create, and Art Laboratory builds it as an editable 3D scene. From there, you can keep describing changes or work directly with the objects yourself, moving, rotating, resizing, recoloring, and arranging them however you like.

One of our favorite features is animation. You can take the scene you've created and bring it to life, turning a collection of static objects into something that moves and feels alive. It's especially satisfying to see something you just imagined become an animated 3D scene.

We also included speech-to-text input to make the tool more accessible. Users can describe their ideas and changes out loud instead of having to type everything.

The goal isn't to have AI make everything for you. It's to remove some of the barriers to 3D creation while still giving you the freedom to experiment, edit, and make something your own.

How we built it

We built Art Laboratory using React and TypeScript, with Three.js for rendering and interacting with 3D scenes. We use Google's Gemini models to interpret natural-language instructions and turn them into structured scene data that our editor can work with.

Rather than generating a single finished image, we represent scenes as individual 3D objects with properties that can be modified. This lets us support both AI-assisted editing and direct manipulation within the same workspace.

We also built an animation system so scenes aren't limited to static objects, along with a backend for handling model requests and validating the results.

Challenges we ran into

A major challenge was making natural-language generation work reliably with an actual 3D editor. Generating objects is one thing, but getting their positions, proportions, colors, and relationships right is much harder, especially when a scene contains lots of smaller details.

Editing existing scenes was also tricky. The AI has to understand what's already there and make the changes you ask for without unnecessarily changing other parts of the scene.

Animation added another layer of complexity. We needed objects to move and behave as intended while keeping everything consistent with the rest of the scene. Getting from a description of an animation to something that actually looks good in motion took a lot of experimentation.

We also ran into challenges with response times, handling unexpected AI outputs, and keeping the editor intuitive as we added more features.

Accomplishments that we're proud of

We're especially proud of how well the animation works. Being able to create a scene, add details to it, and then watch the whole thing come alive is probably one of the most rewarding parts of using Art Laboratory.

We're also proud that the results aren't just something you look at. You can select objects, modify them, add new ones, and continue building on what you've already created.

Another important accomplishment was making these features accessible without requiring users to learn complicated modeling workflows first. Between natural-language instructions, speech-to-text, and the visual editor, there are several ways to approach the same creative task.

What we learned

We learned that making AI useful in a creative application takes more than getting it to generate something impressive. The results need to be editable, consistent, and easy to build on.

We also learned a lot about managing structured 3D scenes, interpreting user instructions, and coordinating animation with a scene that can change at any time.

Perhaps the biggest takeaway was how much the editing experience matters. Even with powerful generation tools, the ability to make small adjustments and experiment freely is what makes the application genuinely useful.