Inspiration

We started out wanting the dream: give our robot arm a prompt, like "stack the blue cup in the white cup", and have it just do it. The popular way to get there is a VLA (vision-language-action model): one neural network that takes in camera images and an instruction and outputs motor commands.

So we tried open-source VLAs on our SO-101. They didn't work. The arm was inconsistent, missed grasps, and couldn't reliably finish tasks it had been prompted to do. Digging into why, we kept hitting the same walls:

Open-source VLAs don't have enough data yet. They're trained on a limited set of robots and scenes, and that doesn't transfer to an arbitrary robot on an arbitrary table.
To be good, a VLA has to be fine-tuned on your robot, or one extremely close to it: same arm, same cameras, similar setup. It requires thousands of hours of demonstrations, lots of compute, and matching training data for a specific hobby arm is hard to come by.
Fine-tuning in simulation doesn't fix everything Many VLAs are trained or fine-tuned on simulated robots, and the sim-to-real gap is very real: lighting, friction, backlash and calibration all differ, and moving a policy from sim to the real arm is its own long project.
When it fails, you can't fix it. A VLA's behaviour lives in millions of weights and there's no ability to understand why it may have missed.

So, instead of training the robot's behavior , what if we wrote it, as short Python programs over a few tested motions, and let a coding agent fix those programs whenever the robot fails? A robot that automatically fixes itself the way a software engineer fixes a bug: read why it failed, change a few lines, test the change in simulation, ship it. It requires no dataset, no GPU cluster, and no sim-to-real transfer of a policy. That's exactly the kind of work Factory Droid is built for.

What it does

Skillsmith is a real SO-101 robot arm whose skills are short Python programs, and Factory Droid rewrites those programs when the arm fails.

All actions the robot takes are emergent behaviors, that the robot developed entirely on its own in under 30 runs.

Try. The arm runs a skill from its library, e.g. stack(top="blue cup", bottom="white cup"), nesting one paper cup into another the way they come in a sleeve. Skills can only call six tested primitives: move_to, grasp, release, look, home, align_over.
Watch. A calibrated overhead camera keeps a live 3D model of the table at 10-ì15 Hz. The attached wrist camera validates grasps.
Classify. Plain rules (no AI) split the attempt into phases (approach, grasp, lift, carry, place, release) and label the failure with numbers, e.g. SLIP_IN_CARRY at call #8: gripped 1.2 cm below an 8 cm rim. Environment problems such as a bumped camera or an object out of reach get directed to humans and never to Droid.
Repair. The robot's own failure automatically triggers Droid through the Droid Python SDK. Droid gets an evidence package: the failure class, a phase table, annotated camera key frames, the motion trace with the failing line, earlier rounds, and lessons from past fixes. It reads the skill, writes a fix, validates it, and tests it in MuJoCo, all through tools we wrote. We operate Droid with a model ladder based on the repair round:
1 claude-haiku-5-5 (Haiku 5.5) effort: medium
2 claude-haiku-5-5 (Haiku 5.5) effort: high
3 claude-sonnet-5-5 (Sonnet 5.5) effort: medium
4 and later claude-opus-5-5 (Opus 5.5) effort: medium
Prove. We don't operate solely on Droid. We re-run the static check, the MuJoCo simulation and the safety envelope ourselves.
Approve. A human approves on the dashboard before the real arm moves.
Remember. The fixed skill is saved as a new version with Droid's one-line lesson ("Grasp tall cups at detected mid-height rather than a fixed distance below the rim"), and later skills like tower reuse it.

How Droid is used (Best Use of the Factory Droid SDK)


Trigger	When we detect a failure using our checks, we automatically trigger Droid.
Control	An explicit tool allowlist. Droid can change exactly one file, `skill.py`, in a workspace outside the repo, and call our tools: `read_file`, `write_skill`, `validate_skill`, `simulate_skill`, `get_scene`, `get_frame`, `get_object_track`, `compare_with_success`, and library lookups. Droid develops the skill and the robot executes it.
Visibility	Every Droid message, tool call, permission decision (allowed or refused, with the reason), diff, and structured result (`diagnosis`, `change`, `change_type`, `confidence`, `lesson`) is an event in `runs/<id>/events.jsonl`, streamed live to our dashboard
Memory	One SDK session per run stays open across repair rounds, so Droid remembers what it already tried. If the same failure repeats after a parameter-only change, the next prompt tells it to change the approach.
Cost and speed	A model ladder starts every repair on Haiku 5.5 at medium reasoning and steps up to Sonnet and then Opus only when a fix doesn't hold.

How we built it

Hardware. An SO-ARM101 (follower plus leader arm, 12 Feetech STS3215 servos), a MacBook Pro running everything, an iPhone as the overhead camera through Continuity Camera, and a USB wrist camera. robotd is the only process that owns the serial port and cameras; it enforces joint limits, a per-step clamp and a watchdog as the last safety layer.

Motion. Top-down IK with placo on the SO-101 URDF. The same kinematics run in the MuJoCo simulation and on the real arm, so the sim gate checks the IK we actually use.

Perception. Checkerboard intrinsics plus ArUco markers whose robot-frame positions we measured by touching them with the gripper tip. Each detection's pixel becomes a ray that we intersect with the plane at the object's known height, which removes the ~2 cm parallax a flat homography leaves on a cup rim:

$$ \mathbf{p} = \mathbf{o} + t\,\mathbf{d}, \qquad t = \frac{h - o_z}{d_z} $$

Detectors: YOLO-World, an HSV backup, and a model-free cup detector (empty-table difference plus rim circle fit). On the real rig, cup positions came in at 0.27 cm mean error, 0.52 cm worst against taped marks. The world model tracks stable object IDs, lets the held object follow the gripper, and raises WorldChanged if anything is bumped mid-skill.

Safety. We run 9 independent layers to verify our work: Droid's tool allowlist, an AST validator on every skill, a restricted runtime, a Cartesian guard before every motion (workspace box, reach, speed cap, call budget), the MuJoCo gate, human approval, robotd's joint clamps, a live bump stop, and failure routing that keeps setup problems away from Droid.

How the four of us built it so fast. We used coding agents to build the robot's coding agent. We split the system into 7 workstreams behind typed contracts, turned every task into a GitHub issue, and ran a shared agent loop on each teammate's laptop: claim an issue, have Claude Code or Droid build it in its own git worktree, pass make check, open a PR that auto-merges when CI is green. CI also enforces file ownership per workstream so agents never collide. 92 PRs merged, 114 issues closed, 1,000+ tests. Humans did the hardware, the calibration and the integration.

Results

Measured with real Factory Droid through the SDK on a seeded-bug benchmark (rim grip, missed grasp, low travel height, wrong place axis, unsafe target, label typo, and more):


Bugs fixed	8 / 8
Fixed in one round	8 / 8
Median time to fix (Haiku 5.5)	14.3 s
Demo failure: fail ‚Üí Droid fix ‚Üí success	25.7 s (test world, before arm motion time)
Sandbox escape attempts blocked	5 / 5

Droid was never told where the bug was; it got only the failure class and a generic hint.

Challenges we ran into

Our first approach didn't work. We spent the start of the hackathon trying to get open-source VLAs to complete prompted tasks on our SO-101, and they were too inconsistent to use without a long fine-tuning effort on our exact robot. Pivoting to code-as-policy meant throwing that work away and rethinking the whole architecture mid-hackathon.
The sim can't see a slip. MuJoCo welds the object on grasp, so it can't predict a cup sliding out of the jaws. That gap is exactly why the camera check exists, and it became the demo's story.
Perception decides everything we can know. A bumped phone camera produces confident wrong numbers. We added a confidence gate: if markers drift or detections get shaky, the failure is labelled environment, not skill, and Droid is never asked to "fix" good code.
URDF vs. real calibration. Joint signs and zero offsets had to be verified against the real arm before we trusted any IK.
A hobby arm. The SO-101 has backlash and 5‚Äì10 mm repeatability, so we designed cup-sized tasks with centimetre tolerance and closed-loop alignment from the wrist camera.
Paper cups are light and flexible. Gripped near the rim, a paper cup slips or squashes; that's the honest failure our demo starts from.
Speed on stage. A repair round has to land in under a minute. A model ladder (Haiku first), a warm SDK session and "smallest edit that fixes the stated failure" got us to ~15 s.
Our own hardware moved mid-hackathon. We started on an NVIDIA GX10 and moved everything to a single Mac with an iPhone camera when the network and drivers fought us.

Accomplishments that we're proud of

Droid does real work in the running product: without it, the arm can't learn a new
Everything the robot's done is completely learned showing emergent behavior.
A permission model we can prove is tight, live, with the real SDK.
Every step the agent takes is visible and replayable.
Sub-centimetre perception on a phone camera and a hobby arm.
A four-person team shipping 90+ PRs in a weekend by orchestrating agents with contracts, CI and file ownership.

What we learned

General-purpose VLAs aren't there yet for most robots. Open-source models lack the data to transfer to a robot they weren't trained on, fine-tuning on your own arm costs lots of demos, compute and time, and sim-trained policies still have to cross the sim-to-real gap.
Code is a better policy format for agents than weights. A three-line diff can be read, simulated, safety-checked and approved; a weight update can't. And a fix takes seconds, not a retraining run.
Classify before you prompt. Giving the agent the failure class, phase and numbers up front is what made one-round fixes possible.
Trust, but verify the agent. Every claim Droid makes ("I tested it in sim") we re-check ourselves.
Agents can write code that passes tests and is still wrong. Behavioural acceptance tests and early hardware integration caught what unit tests didn't.

What's next

Harder benchmark cases that need the camera frames to diagnose, to measure what the visual evidence adds.
Droid-written monitors that watch a skill while it runs, not just after it fails.
Self-calibration: fit repeated position errors into calibration offsets automatically, so Droid only ever fixes logic.
More skills built on top of saved ones: towers, sorting by colour, putting objects in containers.