Inspiration
Clay is one of the first things people ever make something with, and you need no skill to start: you just press it. Most 3D tools hide that feeling behind menus and controllers. I wanted it back, in my own room, using the hands I already have.
What it does
hAnd bUilder is a mixed reality sculpting studio for Meta Quest 3. It opens from a link, with no controllers and no install, and you shape clay anywhere in the air of your room.
- Each finger has a job: thumb + index adds or carves, middle pulls, ring smooths, and the right little finger paints.
- Press the clay with your open hand and it flattens under your palm.
- Show 1, 2 or 3 fingers on the left hand and the right hand places a box, cylinder or cone.
- A right fist grabs what is at your hand. A left fist grabs the whole piece. Two fists turn and resize it.
- Select areas or strokes, move or copy them, and REPEAT the move to build arrays.
- Left thumb + little finger opens the tool pad, which can be laid on a real table.
- Mirror and radial symmetry, undo and redo, colours and materials from pearl to gold and chrome.
- A PC with a webcam and a phone can join with a code and show the live sculpt as extra camera views.
How we built it
WebXR with Meta's Immersive Web SDK and three.js, running in the Quest browser. The clay is a sparse voxel signed distance field. Meshing runs in Web Workers so the hands stay responsive, and resting parts of the sculpture are merged to keep draw calls low. Plane detection finds real surfaces for the tool pad. The webcam view estimates its position by matching hands seen by the camera with the hands tracked by the Quest. Live viewing uses peer-to-peer connections. Built with Claude Code as coding partner.
Challenges
Keeping a steady frame rate on a standalone headset while a sculpture grows. Testing on the real Quest showed draw calls were the bottleneck, not triangles. Hand gestures also had to stay separate, so a hand closing into a fist does not fire a pinch.
What we learned
Test on the real headset early. The emulator said everything was fine; the Quest told the truth within a minute.
What's next
Target launch: Q1 2027. Planned: persistent anchors, more brushes, export to 3D printing and other 3D tools, and sculpting together in the same piece.
Built With
- javascript
- meta-quest
- three.js
- webxr