BigFluffyRobot

Inspiration

We were at the BracketBot workshop staring at a 16-DOF robot with two 7-joint arms and a meter-long telescoping mast on each side, and someone said "it kinda looks like a boxer". That was the whole origin. The arms were already the right reach (about 0.72 m), nearly the same as a human arm and the masts could drop the entire torso almost a full meter. If we could get it to mirror a person's arms in real time, we'd basically have a sparring partner.

Most teleoperation feels like remote control; we wanted embodiment. Instead of pushing joysticks and waiting for a reaction, your body becomes the interface. Raise your fist, the robot raises its fist. Duck, it ducks. Just a webcam, near-zero latency, and complete 1:1 movement.

What it does

BigFluffyRobot turns a BracketBot into a remotely operated boxing robot controlled entirely through body tracking:

Puppet mode: stand in front of your laptop camera and the robot mirrors your arms in real time at 30 fps. Straighten your arm and the robot's arm straightens. Bend your elbow and the robot bends its elbow the same way. Crouch and the mast drops the whole torso. Twist your shoulders and the wheeled base rotates to match, one-to-one.
Record & replay: press Record (with an optional countdown so you have time to get into your stance), perform a combination, and it saves as a named move you can replay on demand. Moves are just small JSON files, so they survive restarts and travel with the repo.
Web dashboard: a browser-based control panel with WASD driving, cartesian nudges for the end-effectors, preset poses (guard, jab, hook, rest), a live 2D robot preview from three angles (side, front, and top-down), and real-time telemetry showing IK error, pose FPS, mast position, and heading.
Grippers: open and close your fist and the robot's gripper follows, tracked by the hand model at 6 Hz. A clenching fist reads as "boxing" instantly.

How we built it

The whole stack is Python on the backend and vanilla JS in the browser, with zero framework dependencies.

Kinematics from scratch. The robot's arms are not anthropomorphic — seven joints, none of which line up with a human shoulder-elbow-wrist. So there's no joint-to-joint mapping. We wrote a damped-least-squares IK solver in pure numpy that takes a target position in 3D space and finds joint angles to reach it. One FK call through the arm chain takes 0.14 ms, and a full IK solve for both arms takes about 12 ms — well within the 33 ms budget for 30 Hz.

Limb-for-limb retargeting. The first version just mapped the wrist position, scaled by arm length. The robot never fully extended — it topped out at about 70% of its reach. The fix was to stop thinking about endpoints and start thinking about limb segments. We take the direction of each human limb segment (upper arm, forearm) and lay them end-to-end using the robot's limb lengths. A target built from the robot's own geometry is inside its workspace by construction. This got us from 70% extension to 98%, and it made the robot actually look like it was copying the pose rather than just reaching for the same point.

Elbow tracking. Hand position alone doesn't pin down an arm's shape — you can swing your elbow through a wide arc without moving your hand. We added the operator's elbow direction as a secondary IK objective, weighted below the hand and only active for the first few solver iterations. The hand converges to within a millimeter; the elbow follows within about 5 degrees on average. When the two objectives conflict, the hand wins — because the hand is what lands the punch.

Torso twist from shoulder width. To turn the base, we measure how much your shoulder line has foreshortened in the camera. A shoulder line rotating about the vertical foreshortens as cos(yaw), so acos(apparent_width / calibrated_width) recovers the twist angle from a plain 2D measurement — no depth sensor, no depth estimate. It's accurate to within a thousandth of a radian up to 60 degrees. The sign (which way you're twisted) comes from which shoulder is nearer the camera. The base holds the heading with a proportional controller on its own odometry, so your shoulders and the robot always agree.

Pose detection in the browser. MediaPipe runs in the browser, not in Python. No pip install mediapipe, no OpenCV camera permissions, no CDN dependency at demo time (we vendor the model files). The browser sends landmarks over HTTP; retargeting and IK happen in Python, keeping kin.py as the single source of truth for the robot's geometry.

Challenges we ran into

The reach problem. Our first retargeting approach anchored the arm at the shoulder link — which is actually the mast carriage, 17 cm away from the joint the arm pivots about. The reach envelope was measured in that wrong frame too. Result: a fully extended human arm produced a robot arm at barely half extension, and the last third of your arm travel did nothing. We didn't realize this for hours because the IK error readout was also wrong (we were clamping to a sphere instead of the real reachable envelope). Fixing the anchor point and measuring the actual non-spherical envelope in 128 directions solved both problems at once.

The base never turned. The dashboard appends a {"v":0,"w":0} idle drive command at the end of every POST batch, after the pose data. When each input wrote the base velocity from its own handler, the trailing zero command wiped the torso's turn command thirty times a second. Puppet mode worked perfectly for the arms; the base just never moved. The fix was centralizing all base control into one mixer function called from the 60 Hz control loop, with the keyboard winning ties.

Gripper chatter. Fingertip tracking near the open/close boundary made the gripper rattle. We added hysteresis — a latch with separate close and open thresholds — and dropped the hand model to 6 Hz instead of every frame. Fist detection doesn't need 30 fps, and the pose model (which positions the arms) does.

The countdown problem. You can't press Record and be in your stance at the same time. Without a countdown, every recorded move starts with three seconds of a hand travelling back from the keyboard. We added a configurable countdown (3s / 5s / 10s) with a big number overlay on the camera feed readable from three meters back.

Accomplishments that we're proud of

The retargeting actually feels right. There's a moment when you forget you're controlling a robot and it just feels like the robot is you. That only happens when the pose copying is good enough that you stop noticing it and getting there meant solving a bunch of problems nobody warns you about. The robot's arms have seven joints and none of them correspond to a human joint. We wrote an IK solver from scratch, figured out limb-for-limb mapping instead of endpoint mapping, added elbow shaping so the arm looks right and not just reaches right, and got the whole thing running at 30 fps in pure numpy with no IK library. A straight punch extends to 98% of the robot's physical reach. That number was 70% twelve hours earlier.

Torso twist from a 2D camera. We're genuinely proud of the shoulder-foreshortening trick. MediaPipe's depth estimates are noisy enough that the first version (which read shoulder depth directly) produced maybe 3 degrees of base yaw from a 45-degree twist. The arccos(width / calibrated_width) approach uses only 2D measurements, recovers the angle to a thousandth of a radian, and needs no depth sensor at all. It's the kind of solution that makes you wonder why you tried the complicated thing first.

Zero dependencies beyond numpy. No FastAPI, no uvicorn, no ROS, no IK library, no ML framework on the server side. The HTTP server is stdlib. The IK is 15 lines of damped least squares. Pose detection runs in the browser so there's no pip install mediapipe. The whole thing vendors into ~20 MB and runs fully offline. When venue WiFi went down during demos, we didn't notice.

Record and replay that actually works on stage. The countdown overlay, the big camera-filling numbers, the fact that the recorder runs on the 60 Hz control loop instead of the browser clock — all of that exists because we tried to demo it once without those things and the first three seconds of every recording were a hand reaching for the keyboard. Small UX decisions like that are the difference between a demo that works and a demo that works in front of people.

What we learned

Writing IK from scratch instead of reaching for a library was the right call — we understood every failure mode, and when the solver did something weird at 2am we could actually diagnose it. The hardest part wasn't the math. It was getting the frames right: which link is the shoulder, where does the arm actually pivot, what coordinate system are the landmarks in. Every bug in the retargeting came down to an anchor point or a sign convention, not the solver itself.

We also learned that the most dramatic feature on this robot is the mast. A 1-metre vertical drop is far more legible on camera than a wheeled base backing up, and it doesn't risk tipping a self-balancing robot. Lead with the duck.

What's next for BigFluffyRobot

Punch dodging. The architecture is already in PLAN.md — time-to-contact estimation from the head camera using the looming equation (TTC ≈ 2A / (dA/dt)), no trained model. Track the opponent's wrist and the growth of its bounding box, trigger an evasive maneuver when TTC drops below 350 ms. The mast makes this dramatic: a 0.45 m duck in 250 ms drops the whole torso out of the way, which reads on camera far better than backing up on the wheels. We had the state machine sketched out (IDLE → ALERT → EVADE → RECOVER) but ran out of hours.

Real hardware. Everything right now runs against a simulated driver that renders in Rerun. The RobotDriver interface is designed so that swapping in a BracketDriver that talks to real motors is one subclass and one line change in main(). We need the motor command API (position targets vs velocity), confirmation that the mast is actually actuated on the physical kit, and a conversation with the balance controller about whether it'll tolerate a 0.45 m mast drop in 250 ms.

Two-player mode. Two operators, two robots, one ring. Each person controls their own robot through their webcam. The dodge system watches the opponent's robot and triggers evasive moves automatically. All the pieces exist — the puppet mode, the threat estimation geometry, the evasive primitives — they just haven't been wired into a head-to-head setup.

Better move library. Right now moves are raw 30 Hz joint recordings. We want to add move blending (transition smoothly between the end of one move and the start of another), speed adjustment (replay a jab at 1.5× without breaking the joint velocity limits), and a combination builder where you chain named moves into sequences with timing gaps.