Virtual Chemistry Lab

Explore chemistry lab with your hands in a virtual lab.

Comment

💡 Inspiration

Chemistry equations describe changes that students cannot always see. We wanted to connect those equations with an interactive experience where students could choose reagents and observe simulated results.

That idea became GestureLab AI: a virtual chemistry laboratory controlled through familiar hand movements using an ordinary webcam.

🧪 What it does

GestureLab AI lets users pick up solid samples, pour liquids into a beaker, and explore chemistry in a first-person 3D laboratory. Webcam hand tracking supports pinch gestures, while mouse controls and a 2D fallback provide alternative ways to interact.

The project includes ten reagents and explicit outcomes for all 45 distinct reagent pairs. Results include equations, observations, and explanations, with a distinction between reactions, no significant immediate reaction, and condition-dependent chemistry.

Visible gas markers help illustrate hydrogen escaping from the beaker. Sodium and water trigger a brief simulated explosion to illustrate a possible hazard. Hazardous combinations display a prominent “Do not try this” warning.

The simulation is qualitative. It does not calculate reaction rates, concentrations, or limiting reagents.

🛠️ How we built it

We built the application with JavaScript and React, using Three.js through React Three Fiber for the 3D laboratory and Tailwind CSS for styling.

MediaPipe detects hand landmarks locally in the browser. Our interaction logic translates those landmarks into pointer movement and pinch actions. Hand and mouse controls share the same system for picking up objects, dropping solids, and pouring liquids.

A deterministic chemistry engine selects the modelled outcomes. The AI supports hand detection, while explicit chemistry rules provide the equations and explanations.

We extended the existing application with a 3D scene while preserving its tracking system, interaction logic, reaction engine, and 2D laboratory.

🧩 Challenges we faced

Preventing accidental pours

Hand tracking can briefly lose a pinch because of movement or lighting. Treating every lost pinch as a bottle release could accidentally add a reagent.

We addressed this by requiring a completed hold over the beaker before a pour commits under hand control. Releasing early cancels the action.

Connecting gestures to a 3D scene

Hand tracking provides screen coordinates, while the laboratory uses 3D positions. We added raycasting to identify the reagent beneath the pointer and a stable plane for carrying objects through the scene.

Making reaction effects understandable

The original bubbles disappeared at the liquid surface, making gas evolution difficult to notice. We added brighter bubbles and gas markers above the beaker. Labels explain that the markers illustrate hydrogen, which is colourless.

We also distinguish a simulated hazard from a guaranteed outcome: the sodium–water burst illustrates a possible explosion rather than predicting one in every case.

Expanding chemistry coverage responsibly

Many combinations initially returned “not modelled.” We expanded the database to cover every distinct pair in the available library, with stated conditions and explanations.

For zinc in alkali, we use a condition-dependent result rather than inventing a fast reaction or claiming that nothing can happen. 📚 What we learned

We learned that reliable interaction requires more than detecting a gesture. The application also needs clear rules for confirming actions, cancelling them, and recovering when tracking is lost.

Separating chemistry from rendering helped us support both 3D and 2D views without duplicating the reaction logic. We also learned to distinguish what a simulation illustrates from what its underlying model actually calculates.

We used an AI coding assistant to help inspect the architecture, diagnose issues, implement changes, and develop checks. Understanding and verifying those changes remain our responsibility.

🏆 Accomplishments

  • Connected webcam hand tracking to an interactive 3D chemistry bench.
  • Preserved mouse controls and the original 2D fallback.
  • Added explicit outcomes for all 45 distinct reagent pairs.
  • Added visible gas effects and reaction-specific hazard warnings.
  • Passed the production build, lint, 214 core checks, and additional coverage and effect-logic checks.

Full browser visual verification and real-webcam testing remain pending. We do not yet claim measured improvements in student learning.

🚀 What’s next

Our next step is to test gesture usability on real webcams and gather feedback from students and teachers. We also want to expand the reagent library and introduce quantitative chemistry, including concentration and limiting-reagent calculations.

Our goal is to help students connect an action at the laboratory bench with an observation and the equation behind it.

Built With

Share this project:

Updates