Airloom

Airloom turns your webcam into a 3D canvas where you can draw, orbit, edit, and assemble spatial artwork in midair using hand, wrist, and head gestures.

Comment 2

Inspiration:

Digital art is often done with a mouse, stylus, controller, or touchscreen. I built a project to remove those tools entirely and turn the space in front of you into a 3D canvas.

What it does:

Airloom is a browser-based 3D painting instrument controlled through gestures.

Pinching your thumb and index finger draws a 3D stroke. Two fingers pan the artwork, while three fingers orbit it and control zoom through hand depth.

With no hands visible, a quick head turn left opens the brush cartridge and a turn right closes it (I recommend first changing the brush size before closing because by moving your hand out of frame while on the color picking, it will change the color on accident). A fist selects colors, while an open palm adjusts brush or eraser size. A fast palm-to-back wrist roll toggles the eraser.

Closing your fist over connected strokes picks them up. You can move them in three dimensions and release near another edge or vertex to snap them together.

After the first stroke, a side viewer appears 90 degrees from the main camera. Raising one index finger temporarily rotates this viewer, which springs back when released. An open-palm swipe left performs an undo action.

Line Smoothing is activated by default and slightly smooths rough or choppy strokes without changing intentional corners or shorter marks. Optional Shape Assist corrects simple geometry like lines, arcs, circles, ellipses, triangles, rectangles, pentagons, and hexagons while keeping the original size, width, orientation, and hand-drawn look.

Airloom also provides keyboard, mouse, trackpad, touchscreen, and visible-button controls when camera mode is disabled or unavailable.

How I built it:

Airloom was built with Next.js, React, TypeScript, Three.js, MediaPipe Tasks Vision, and WebGL.

MediaPipe tracks hand and face landmarks locally in the browser. A custom gesture engine converts landmark movement into drawing, navigation, palette, erasing, undo, side-view, and object-manipulation controls. Hysteresis, hand-relative measurements, smoothing, release windows, tracking-loss tolerance, and low-light correction help turn tracking data into an input we can use.

Three.js converts recorded points into smooth tubular geometry and handles rendering, raycasting, camera transformations, erasing, object movement, and geometric snapping. A second camera renders the live side view.

Shape Assist uses normal geometric recognition to identify simple shapes and correct them while preserving the scale and general look of the original stroke.

Challenges:

Raw hand tracking is not the best, it can briefly lose fingers, especially during fast movement or in low light. Early versions delayed strokes, cut continuous lines apart, and confused different gestures. I addressed this with gesture priorities, separate activation and release thresholds, motion-aware smoothing, short tracking-loss tolerances, and low-light correction.

Just drawing in 3D was another challenge because a webcam provides limited depth information. The side viewer, depth-based movement, object repositioning, and snapping system help artists understand and correct spatial placement.

I also had to transform drawing and erasing coordinates whenever the artwork was moved or rotated so the cursor remained aligned with the visible canvas.

Making everything work properly on phones and tablets was another challenge. I had to create separate touch controls and layouts instead of simply cramming the desktop interface onto a smaller screen.

Accomplishments:

Airloom became a complete creative instrument rather than a hand-tracking demonstration (that was my original idea). It supports spatial drawing, 3D navigation, adjustable tools, erasing, undo, line smoothing, shape correction, object movement, depth correction, snapping, touchscreen controls, and exporting.

All camera processing and gesture recognition happens locally in the browser. No captured video leaves the user’s computer.

What I learned:

Gesture detection is not only about recognizing poses. I learned that they also require understanding transitions, timing, motion, confidence, lighting, and context.

What’s next for Airloom:

Future additions could include project saving, layers, custom brush materials, 3D model export, collaboration (especially in the same document, a Figma inspired idea I really wanted to add), and calibration that adapts gesture thresholds to each user.

Built With

Share this project:

Updates