VoiceScene

Inspiration

Building 3D scenes usually means clicking through menus or writing code - I wanted to see if you could just describe what you want out loud and have it appear. The idea was to make scene creation feel like directing rather than programming.

What it does

VoiceScene lets you build and edit a 3D scene entirely by voice. Hold a push-to-talk button, say something like "add a red cube" or "make the sphere spin," and the scene updates live. A JSON scene-graph panel next to the 3D view shows exactly what changed, so the underlying state is never hidden.

How I built it

The architecture is built around a single JSON scene graph as the source of truth - objects, camera, and lighting all live in one structured state. A diff-renderer in three.js updates only what changed between the old and new scene, rather than re-building the whole scene every time.

Voice input uses push-to-talk with the MediaRecorder API, sending audio to OpenAI's Whisper API for transcription. The transcript, along with the current scene state, is then sent to GPT-5.6 (gpt-5.6-luna), which is given a fixed set of tools - add_object, modify_object, remove_object, set_camera, animate, and stop_animate - rather than being allowed to generate arbitrary code. This keeps the system reliable: GPT-5.6 can only make changes the renderer already knows how to handle safely.

I built this almost entirely inside Codex CLI, using GPT-5.6 as the coding agent. I worked through the project in stages: first scaffolding the Vite/three.js project and the scene-graph types and diff-renderer, then wiring up push-to-talk and Whisper transcription, and finally building the GPT-5.6 tool-calling agent loop that ties voice commands to scene changes.

Challenges I ran into

The main technical challenge was making the agent loop robust rather than just working once. Early on, GPT-5.6's function-calling wasn't compatible with the reasoning_effort default on the chat completions endpoint - fixed by explicitly setting reasoning_effort to 'none' for tool-calling requests. I also had to make sure the agent asks for clarification instead of guessing when a voice command is ambiguous, so it never silently makes the wrong change to the scene.

What I learned

Constraining an LLM to a fixed tool schema, rather than letting it emit raw code, made the whole system dramatically more reliable - bad or ambiguous input degrades to "ask a clarifying question" instead of crashing the render. This felt like the right way to build any voice or language-driven interface on top of a live, stateful system.

What's next

Expanding the vocabulary of supported shapes, materials, and spatial relationships (like "next to" or "behind"), and adding undo support so voice-driven mistakes are easy to correct.

Built With

• codex
• express.js
• gpt-5.6
• node.js
• openai-api
• three.js
• typescript
• vite
• whisper-api