Teleoperation-Arm

Mimic: your webcam is the controller
"I'm your arm now." The 3D twin mirrors your shoulder, elbow and wrist live
Wrist twist and hand tracked by MediaPipe, sent to the SO-101 in real time
Lost tracking? It pauses until your hand lines up with the robot again

Inspiration

A surgeon is stuck in traffic. A specialist is three states away. The patient can't wait.

Distance shouldn't be the reason someone doesn't make it. We kept coming back to one question: what if the only thing a doctor needed to help was a camera? No trip to the hospital, no expensive console. Just get on a call and use your own hands.

That's where Mimic started.

What it does

Mimic turns an ordinary webcam into the controller for a real robot arm (the SO-101). Stretch your right arm straight out and the robot syncs to you. From then on it copies your shoulder, elbow, wrist twist and hand as you move.

If it loses sight of you, it doesn't guess. It freezes where it is and waits until your hand lines up with the robot's again. No leader arm, no gloves, no joysticks.

How we built it

MediaPipe tracks your arm and hand right in the browser and turns them into angles for the robot's six motors. A Python bridge on top of LeRobot sends those angles to the arm over USB, and a three.js twin shows every move on screen so you can see what the robot is about to do. Gemini turns typed commands into preset gestures, and ElevenLabs gives the robot a voice.

Technical details

Pose pipeline: MediaPipe pose and hand tracking in the browser, using the heavy pose model (served locally) and 3D world landmarks so reaches toward and away from the camera are measured correctly. Tracking locks onto the operator's right arm and holds it even when MediaPipe swaps its left/right labels.
Joint mapping: landmarks map to the six motors (shoulder pan and lift, elbow, wrist flex and roll, gripper) angle for angle, with joint directions taken from the SO-101's URDF.
Inverse kinematics: exact SO-101 kinematics mirror the hand's position, and how open your hand is drives the gripper through its full range.
Calibration and safety: hold your right arm straight out to sync. The robot saves a start pose, returns to it on load or reset, and pauses when tracking is lost until your hand lines up with the robot again.
Python bridge: a WebSocket and HTTP server on top of LeRobot's Feetech driver streams joint goals to the arm over USB and sends telemetry back, with rate limiting, automatic reconnect and a dry-run mode.
Digital twin: a three.js model of the real SO-101 shows live joint state next to the target pose.
Voice and commands: Gemini turns typed commands into preset gestures from a fixed list, and ElevenLabs gives the robot a voice. API keys stay on the bridge, never in the browser.
Testing: a Node UI regression suite checked against a baseline commit, plus a video test source so it runs without a webcam.

Challenges we ran into

A webcam sees a flat picture, but your arm moves in 3D. Reaching toward the camera and reaching sideways can look almost the same, so we switched to MediaPipe's heavy model and its 3D landmarks to tell them apart.

MediaPipe also kept swapping our left and right arm mid-move, and the robot would jump to the wrong side. The arm dropped off USB more than once. Our first sync step was so strict we could barely get it to lock on. And late in the night we'd piled on so many features that we rolled back to an earlier version, because it simply moved better.

Accomplishments that we're proud of

The moment the arm first followed one of us across the room, from nothing but a laptop camera. It moves when you move, pauses when it should, and picks up right where you left off.

What we learned

Tracking points on a body isn't enough. The robot has to understand how an arm is built: which joints move together, how far each one can go, and when it's safer to stop than to guess. Simple and reliable beat clever every time.

What's next for Teleoperation-Arm

We're a long way from an operating room, and we know it. Next is better 3D tracking, phone cameras, finer wrist and finger control, and stronger safety checks. After that, control over the internet, so the person moving the arm and the arm itself really can be in different places.

Today it's a weekend project. We think it's a small step from "you need to be there" to "you need a camera."