Inspiration
Chemistry is the subject students most often learn without doing it. Many schools can't run real labs: chemicals cost money, some are toxic (lead iodide, the classic bright-yellow precipitate, is one), and there's never enough lab time. So students memorise "PbI₂ is yellow" from a worksheet instead of seeing it. Teachers find out what a class misunderstood only after the test.
We asked: what if the worksheet a teacher already uses became a lab students could actually do, with their own hands, safely, on any laptop with a webcam?
What it does
SparkLab is a 3D chemistry lab you control with your hands, no controller, no VR headset.
- Walk into the lab. You start in a 3D lab building. A speech bubble over your student asks "Which one do you want to learn today?" Point at Organic Chemistry, Inorganic Chemistry, Acids & Bases, or the Experiments Lab, and your student walks there and asks "Ready for questions?"
- Build molecules by bonding atoms. "What is the formula of water?" Atoms float around you. Close your fist on an atom to grab it, grab another with your other hand, and bring your hands together: H + H → H₂, then H₂ + O → H₂O, which snaps into its real bent shape. Try to add carbon and it flashes red: "Carbon isn't in water." Add a third hydrogen: "Water has only 2." Every count is done by code.
- Pour, and watch the reaction. Close your fist on a beaker, hold it over the flask and tilt your hand: the beaker tips with your real wrist, a stream pours in, the flask fills. Mix lead(II) nitrate and potassium iodide and a bright yellow solid rains down; mix acid and base with phenolphthalein and it turns pink. Then an explanation card shows the balanced equation and why (ions swap, PbI₂ is insoluble, K⁺ and NO₃⁻ are spectators).
- Experiments Lab: 7 guided experiments, step by step: volcano eruption, elephant toothpaste, balloon inflator, copper-plated nail, red-cabbage rainbow, lava lamp, hot & cold packs. For the volcano, code computes the moles, the limiting reactant and the litres of CO₂ from exactly what you poured.
- Snap a real worksheet. A photo of a chemistry worksheet becomes playable questions. AI agents read it, solve it and predict the mistakes students make; then code recomputes every answer, balances every equation and checks every reaction against solubility rules before a student sees it.
- An AI coach adapts after every answer, can write a new question aimed at your exact mistake, and code re-checks it before it's shown.
- The teacher sees mistakes live on their phone (QR code, no login): each answer appears within seconds with the mistake named, plus the class's top mistakes.
How we built it
- Next.js 16 + TypeScript + React Three Fiber for the 3D lab, the building, the bench scenes and the effects (pouring stream, falling precipitate, bubbles, colour changes).
- Hand tracking that runs entirely in the browser: MediaPipe Pose (where your hands are) plus MediaPipe Hand Landmarker (21 points per hand) tell an open hand ✋, a pointing finger ☝️, and a fist ✊ apart, plus the hand's tilt for pouring. We designed the rules so a resting open hand never selects anything: point and hold to choose, close your fist to grab. Camera frames never leave the laptop.
- We tuned the gestures from real recordings. We built a landmark recorder (landmarks, not video) and recorded 13 clips: clapping, raising both hands, aiming at 30/45/60°, drinking water, scratching your head, talking with your hands. Replaying them showed, for example, that our "both hands up" rule fired when you dipped your head while clapping. We fixed the thresholds from the data, and every clip is now an automated test.
- 3D models generated with Meshy AI from text prompts in one consistent style (lab benches, fume hoods, shelves of reagents, Bunsen burner, stools, microscope). Glassware, liquids and molecules stay in code because their colours change live.
- Gemini powers the AI agents through one wrapper with timeouts, retries, schema validation and a log of every call: a vision Reader-Solver, a Mistake Modeler that writes a wrong formula for each misconception, a Coach, and Teacher Insights.
- AI writes, code checks every number. The AI only fills draft schemas; our code builds the final question. A chemistry parser balances equations (every arrow style, hydrates, parentheses), solubility rules check precipitates, and molecule counts come from parsing formulas. No number a student sees comes straight from the model.
- MongoDB Atlas stores worksheets, sessions, attempts and every AI decision; one aggregation powers the live teacher view.
- Deploying on Vultr. We planned to host SparkLab on a Vultr Cloud Compute server: we created the account, redeemed the MLH credits, and wrote a one-command deploy script that installs Node and Caddy, builds the app, runs it as a service, and serves it over HTTPS with a free certificate (the webcam only works on HTTPS). But due to internal error at Vultr, we couldn't create the server before the deadline. As a fallback we served the app through a Cloudflare tunnel; the Vultr deploy is ready to run the moment the account is approved.
Challenges we ran into
- Gestures for a seated student in a noisy hall. Pose tracking alone can't tell a fist from an open hand, so early versions picked answers whenever a hand rested on them. Adding finger tracking, and making "open hand = never select", fixed it.
- Debugging motion is hard without data, so we built the clip recorder and turned real recordings into tests. It caught bugs we would never have seen by hand.
- Trusting AI in front of students: a two-layer schema where code computes every answer, a repair loop that sends code's exact errors back to the model, and a coach whose own questions get re-checked.
- Making it feel physical: pouring needed the beaker to follow your real wrist tilt, the stream to only flow when you're over the flask, and the flask to fill.
Accomplishments that we're proud of
- You can build H₂O by bringing two hands together and get chemically correct feedback at every step.
- A real pour-and-watch lab for reactions many classrooms can't safely run (it's virtual only, and we say so on screen).
- Gesture rules backed by recorded data and automated tests, not guesswork.
- A safety-first design: no video leaves the device, and hazardous reactions carry a "virtual only" warning.
What we learned
The hard part of a hand-controlled app isn't tracking hands, it's deciding what a hand means. A resting hand, a pointing finger and a grab feel obvious to people but needed careful rules, and real recorded data, before the computer agreed. We also learned to let AI do the creative work (reading worksheets, predicting mistakes) while code owns every fact.
What's next
Titration with an open-hand burette valve and a live pH curve; a "mystery sample" identification lab (flame tests and precipitates); saving lab results to the teacher's dashboard; VSEPR shapes in the molecule builder; and a classroom mode where students race to build the same molecule.
Built With
- cloudflare
- gemini
- google-ai-studio
- llm
- mathjs
- mediapipe
- meshy-ai
- mongodb-atlas
- nextjs
- react
- react-three-fiber
- tailwindcss
- three.js
- typescript
- vultr
- web-speech-api
- zod
- zustand