GestureLab

Inspiration

Having navigated through 3D modeling software, we noticed that the user experience is very clunky and unfriendly, with a steep learning curve to use efficiently, due to the software doing its best to turn 2D mouse movements combined with keyboard shortcuts into 3D movement.

What if, instead of using the mouse and keyboard to interact with objects in the workspace, we could use our hands and directly manipulate the entities? This would not only completely bypass the challenge of maneuvering a 3D space with 2D inputs, but it could also prove to be more accessible to people with disabilities, especially with our addition of voice recognition.

What it does

Gesture Lab is a browser-based 3D modeling tool that relies on input from your hands and voice, using nothing but a webcam. Workspaces are able to be saved and loaded as .json files on the device.

Hand Gestures Pinch to grab: pinch an object with your thumb and pointer finger to pick it up and move it. Thumb + index will snap the object to the grid or flush with other objects. Thumb + middle will allow you to move the object freely. Push and pull for depth: move your hand closer to the camera to move an object deeper into the space and pull it away from the camera to bring it closer. Two-hand scale: pinch both hands and bring them apart or together to resize. Two-hand rotate: twist both hands like a steering wheel to turn an object, or move both hands up or down to tilt it. Move the camera: pinch empty space and move to rotate the view. Push or pull to zoom in or out. Press buttons: hover over a button and pinch to press it. The entire app can be run without a mouse.

AI Assistant Using the microphone button, you are able to voice out commands, with Claude turning those requests into actions on the screen. Using this, you are able to add objects to the workspace and adjust their orientations.

How we built it

Gesture Lab is composed of 3 different services working together in real time:

Computer vision backend (Python): OpenCV captures the webcam, and MediaPipe Hand Landmarker tracks 21 landmarks on each of up to two hands. We detect pinches with hysteresis to avoid flicker, smooth hand positions, detect two-hand scaling, and stream everything to the browser over Flask-SocketIO.

AI backend (Node.js / Express): Claude (Anthropic API) runs as a tool-using agent with tools for creating, moving, rotating, scaling, recoloring and deleting objects, plus generating models. It receives the current scene each turn (including which object you have selected), so it can resolve “it” and “that sphere”. Model generation calls Fal’s Hyper3D Rodin text-to-3D and returns a .glb.

Frontend (React + TypeScript + Vite): Three.js through React Three Fiber and drei renders the scene, and Zustand holds the shared scene state, so gestures, the AI, the gizmo and the Inspector all edit the same objects. All gesture interpretation happens in the frontend from raw landmarks: raycasting from the hand cursor into the scene, camera-facing drag planes, depth from apparent hand size, snapping, collision push-out, and the scale/rotate mode lock. Voice uses the browser’s built-in Web Speech API.

Challenges we ran into

Telling a twist from a stretch. Our first iteration of the two-hand rotation mostly resized objects instead. The cause: the webcam image is 4:3, and distance measured in normalized coordinates grows when hands rotate toward vertical, so a twist looked like a 33% stretch. Correcting for the aspect ratio, plus “first clear movement wins” mode locking, fixed it.

Noisy, ambiguous hand data. Raw landmarks jitter, hands disappear from frame, and MediaPipe’s left/right labels can swap. We added pinch hysteresis, smoothing, deadzones, release safeguards (letting go when a hand vanishes or the connection drops), and tracking by position rather than by label.

Depth from a 2D webcam. A single camera has no real depth, so we used the hand’s apparent size as a proxy: reaching toward the screen makes your hand look bigger, which pushes the object deeper, and vice versa.

Accomplishments that we're proud of

A multimodal interface: hands, voice, text, and mouse all edit the same scene, allowing you to switch between them at any time.

Text to 3D in the scene: being able to describe an object, see it be created, and then "physically" pick it up feels like magic.

Being able to create an entire scene without a mouse: create with your voice, and manipulate objects with your hands and voice, no mouse needed.

What we learned

Gesture design is more about what to ignore, rather than detection. Through deadzones, hysteresis, and mode locks, we were able to prevent sensor noise and consequently unwanted gestures/inputs.

Small geometry details, such as aspect ratios, coordinate spaces, and which plane you drag on can make or break how a gesture unfolds.