-
Logo
Inspiration We built NeuroForge because neuroanatomy is notoriously difficult to learn from flat 2D textbook diagrams. As medical students, we kept wishing we could just talk to a brain model or wave our hands to spin it and inspect a region. We wanted to make MRI exploration as natural as flipping a physical model on a desk — but powered by AI.
What it does NeuroForge is an interactive 3D brain explorer that lets users navigate neuroanatomy using voice commands, hand gestures, and a Gemini-powered AI assistant. You can speak a brain region (e.g. "show me the frontal lobe") and the 3D model automatically orients itself to highlight it. Pinch to drag and rotate the brain in real-time; release to select the region you're targeting. A built-in chatbot answers questions about anatomy, function, and neuroimaging on the fly.
How we built it We built the frontend in React + TypeScript with TanStack Start and rendered the 3D brain with Three.js inside a custom InteractiveBrain component. Hand tracking runs via MediaPipe, where we detect pinch vs. open-hand states to handle drag/selection logic. Voice control pipes speech through the Web Speech API into a command parser that maps region names to camera rotation targets. The Gemini chatbot streams through a createServerFn endpoint using Google's generativelanguage API with a neuroscience-tuned system prompt.
Challenges we ran into Getting the camera auto-rotation math right was surprisingly tricky. Three.js Euler order meant our yaw/pitch signs kept flipping the brain away from the target region instead of toward it. We also iterated heavily on the gesture model: we tried fist detection, then switched to a "pinch-to-drag, release-to-select" flow after testing what felt most natural without a controller. Balancing real-time hand-tracking performance with smooth 3D rendering took a lot of tuning.
Accomplishments that we're proud of We're proud that the whole interaction loop feels invisible — you speak or move your hand, and the brain just responds. Getting the voice-to-camera-rotation pipeline working end-to-end, with accurate region highlighting, was a huge win. We also love that the Gemini assistant genuinely knows neuroanatomy; it doesn't just regurgitate Wikipedia, it contextualizes answers for medical students and clinicians.
What we learned We learned a ton about Three.js coordinate systems and how tricky intuitive 3D UX really is. Small tweaks in gesture cooldown timing or rotation interpolation make or break the experience. We also learned how to structure a full-stack AI feature in TanStack Start, from client chat UI to serverless function calling the Gemini API securely.
Log in or sign up for Devpost to join the conversation.