Inspiration
When you describe a shape, you naturally use your hands to show its size and orientation. We wanted to turn those movements into a way to actually build it.
Handmade brings that idea into Blender using a regular webcam and microphone.
What it does
Handmade lets you create cubes, spheres, cylinders, and triangular prisms with voice commands. You select an object with one hand and use the other to move, rotate, resize, or extrude it.
Everything happens inside Blender, so you’re making an editable 3D model. Undo and cancellation help you recover from unwanted changes.
How we built it
We built a Python companion using OpenCV for webcam capture, MediaPipe for hand tracking, and ElevenLabs for real-time speech recognition.
It sends structured events over a local TCP connection to our Blender add-on, which handles selection, gesture interpretation, previews, and mesh edits.
Our three-person team split the initial work between hand tracking, voice control, and Blender integration.
Challenges we ran into
Similar gestures were difficult to distinguish. Closing a hand to scale could also look like rotation.
Pinching also moved the fingertip away from the user's aim. We adjusted the cursor behavior to preserve that aiming position and use steadier palm movement during a pinch.
Camera permissions, microphone access, background noise, tracking loss, and Blender’s Undo behavior added their own challenges.
Accomplishments that we're proud of
We connected live hand tracking and voice commands to real Blender modeling operations.
We also built previews, cancellations, and individual Undo steps. Getting three separately developed components working together was a big milestone.
What we learned
Detecting a gesture is only the beginning. Deciding when it starts, what it means, and when it ends is what makes the interaction usable.
Clear feedback and shared interfaces mattered as much as the tracking itself.
What's next for Handmade
We’re refining gesture recognition and adding clearer face selection with hover highlighting and deliberate confirmation.
We also want user calibration, finer control over distances and angles, and support for more Blender modeling tools.
We aim to improve the voice recognition module in order to increase the number of actions users can make with their voice.
Log in or sign up for Devpost to join the conversation.