DrawDynamics

A voice-controlled AI physics laboratory that listens, builds, predicts, and explains real-time simulations.

Comment

Inspiration

For university students and autodidacts, there is a frustrating gap in physics education. Flat textbook equations lack intuitive visual feedback, and traditional physics sandboxes (like PhET) require tedious manual clicking and dragging. We wanted to build a bridge: an interactive, real-time physics laboratory where students can simply talk to an AI tutor, ask for an experiment, and watch the math come to life on the screen.

What it does

DrawDynamics PRO is a real-time, multi-agent AI physics sandbox. It allows users to control a 2D physics engine using their voice, predicting outcomes, and dynamically calculating formulas.

Key features include:

Agentic Object Spawning: Users can speak into their microphone (e.g., "Spawn a heavy box colliding with a small ball"), and the AI instantly builds the scenario on the canvas.

The "Predictive Oracle": By analyzing a hidden JSON snapshot of the canvas, the AI can predict the outcome of complex collisions or kinematic events before the user unpauses the engine.

Proactive System Alerts: A background observer constantly monitors the Matter.js engine. If a massive kinetic event occurs (like a high-speed collision), it seamlessly injects an alert into the Live API session, prompting the AI to interrupt and excitedly explain the physics unprompted.

Real-Time Math Log: As the AI explains concepts, a background agent extracts the relevant formulas and renders them dynamically in a LaTeX-powered UI.

Custom Visual Polish: Features glowing tracer trails and live velocity vector arrows to visually prove the math of projectile motion and gravity.

How we built it

We utilized a robust, asynchronous monorepo architecture separating the simulation from the AI orchestration:

Frontend (UI & Simulation): Built with React, Vite, and Tailwind CSS. The physics are powered by Matter.js, with Zustand managing the state. KaTeX is used for mathematical rendering. Deployed on Vercel.

Backend (The AI Core): A containerized Python FastAPI server deployed to Google Cloud Run. It handles a continuous, bidirectional Secure WebSocket (wss://) connection.

The Multi-Agent AI System: We used the Google Gemini Multimodal Live API as the primary voice interface, handling the low-latency audio streaming. Running in parallel, we deployed Gemini 2.5 Flash as two distinct background agents:

The Architect: Parses conversation history into strict JSON commands to spawn or manipulate Matter.js objects.

The Mathematician: Listens to the tutor's explanations, extracts core physics formulas, and pushes them to the frontend.

Challenges we ran into

Agentic Eavesdropping: Initially, our "Architect" agent would hear the user ask for a simulation analysis and accidentally clone/duplicate all the objects on the screen. We had to heavily refine our prompt engineering and system instructions to establish strict boundaries between "building" mode and "analyzing" mode.

WebSocket Protocol Clashes: Deploying to Google Cloud Run introduced a severe 502 Bad Gateway error. We discovered that GCP was forcing HTTP/2 traffic into our Uvicorn server, which broke the WebSocket handshake. We resolved this by reconfiguring the Cloud Run service to enforce standard HTTP/1.1.

State Synchronization: Keeping the AI aware of a chaotic, 60fps physics engine without overloading the API required building a specific "JSON Snapshot" system that only sends data when explicitly requested or when a safety threshold is breached.

Accomplishments that we're proud of

Successfully orchestrating three parallel AI streams (Live Audio, JSON building, and LaTeX rendering) over a single WebSocket connection without blocking the frontend physics engine. The architecture is incredibly stable and feels like magic to interact with.

What we learned

We learned the critical importance of containerization (Docker) and IAM permissions when deploying automated Python environments to Google Cloud. Furthermore, we learned that Large Language Models perform exponentially better when isolated into specific "roles" (like our Architect and Mathematician) rather than trying to make one single prompt do everything.

What's next for DrawDynamics

We plan to expand the educational scope of the application to include thermodynamics and optics. Ultimately, this engine will be integrated into a larger software portfolio to help freelance software developers demonstrate complex AI architectures to potential clients.

Built With

Share this project:

Updates