Inspiration
TinyDirector started with a simple question: What if I could talk to Blender and have it create whatever I described?
At first, I was experimenting with using NVIDIA Nemotron to control a 3D environment through natural language. I could ask it to create objects, change their properties, and manipulate scenes without manually navigating Blender.
But I quickly realized something: making a cube appear when I asked for one was cool, but what was the actual purpose?
That led me to a more interesting question: What if, instead of just creating objects, AI could help people understand things by bringing them to life?
I started thinking about how most AI tutors work. You ask a question, receive a wall of text, and maybe get a diagram. But understanding a concept often comes from experimenting with it and seeing what changes.
That was the turning point. I transformed TinyDirector into Poyo, a conversational 3D tutor that doesn't just explain concepts—it lets you explore them, change variables, and see what happens.
My goal became making learning feel more like experimenting with a patient teacher than reading another chatbot response.
What it does
Poyo turns Blender into an interactive classroom powered by NVIDIA Nemotron.
I can ask, "Why does an LED need a resistor?" and Poyo creates a 3D electrical circuit, calculates its current, animates the flow, and explains what's happening.
Then I can ask, "What if I increase the resistance to 1,000 ohms?" Poyo updates the existing experiment, recalculates the values, and changes the visualization.
I also built a What-If Lab where Poyo can display two versions of a circuit side by side, allowing me to compare the results. It can even take me on a narrated camera tour, highlighting individual components and explaining their roles.
For supported electronics topics, Poyo can generate new lessons using a multi-agent pipeline rather than relying entirely on predefined animations.
I expanded the idea beyond electronics with Spot the Scam, an interactive cybersecurity lesson that teaches users to identify suspicious text messages, explore simulated consequences, and learn safer responses.
Underneath everything is a knowledge base of concepts, referenced facts, and common misconceptions, along with verification checks to help keep Poyo's lessons accurate.
How I built it
I built Poyo around an AI orchestration pipeline combining NVIDIA Nemotron, Blender, Python, SQLite, and OpenAI APIs.
NVIDIA Nemotron is the central director. It interprets requests, selects tools, generates explanations, and creates structured lesson specifications. Its vision capabilities also help inspect generated lesson previews.
Blender provides the actual 3D environment. I built a worker that receives validated instructions, creates or modifies scenes, and reports the resulting objects and properties.
For accuracy, I implemented deterministic Python calculators, JSON schema validation, and Blender scene read-back checks.
For new electronics lessons, I developed a multi-agent workflow where Nemotron plans the lesson, an OpenAI builder generates restricted Blender toolkit code, and an isolated Blender process builds the scene. A verifier checks the resulting lesson before it is loaded into the main environment.
I also created a SQLite knowledge base containing educational concepts, relationships, sourced facts, and misconceptions.
Finally, I added a voice interface using OpenAI speech services so users can communicate with Poyo conversationally while Blender remains the main visual experience.
Challenges I ran into
The biggest challenge was balancing generative freedom, reliability, and speed.
Initially, it was much easier to produce predefined animations. But I didn't want Poyo to become a chatbot that simply triggered prerecorded lessons.
Generating new scenes introduced another set of problems: unpredictable execution times, incorrect wiring, overlapping labels, and inconsistent model-generated instructions.
To address these issues, I combined reusable visual components with runtime lesson planning, deterministic calculations, validation, and a repair-and-verification pipeline.
I also had to solve persistent scene editing. When someone asks to change a resistor, Poyo should modify the existing circuit rather than rebuild everything from scratch.
Synchronizing spoken explanations with Blender's camera movements was another challenge, especially when model responses varied in length.
Accomplishments that I'm proud of
- Built a working voice-controlled 3D tutoring experience using NVIDIA Nemotron and Blender.
- Implemented persistent scene editing and contextual follow-up questions.
- Created a cinematic What-If Lab with animated camera tours and side-by-side circuit comparisons.
- Developed a multi-agent system that generates, builds, verifies, and loads new lessons for supported electronics concepts.
- Built deterministic calculation and Blender scene-verification systems.
- Integrated a structured SQLite educational knowledge base.
- Expanded Poyo beyond electronics with an interactive scam-awareness lesson.
- Added automated testing and fallback mechanisms to improve reliability.
What I learned
My biggest takeaway was that making an AI model generate something is much easier than making that generation dependable.
I learned that reliable AI agents need much more than prompts. They need structured plans, state management, validated tool calls, execution boundaries, error handling, and verification.
I also learned how powerful it can be to combine generative AI with conventional programming. Nemotron handles flexible reasoning and lesson planning, while deterministic tools provide accurate calculations and predictable execution.
Most importantly, I learned that a 3D visualization becomes much more useful when someone can interact with it and ask questions about what they're seeing.
What's next for Poyo
I want to make Poyo faster, expand the subjects it can teach, and support richer forms of interactive simulation.
I'm also interested in bringing Poyo into virtual and mixed reality, where students could physically walk around their lessons, inspect components from different angles, and eventually manipulate experiments using their hands.
My long-term vision is to make Poyo a learning companion that can transform complex ideas into interactive environments that anyone can explore.
