watch2fold

• 

  One human demo in, judged folds out — dual SO-101 arms in RaiSim

• 

  Closed-loop rollout: the policy's own gripper targets drive grasping

• 

  Every number reproducible from the repo; Arm64 verified on each push

• 

  Our physics converged on the same fold technique Memo uses in production

Why this is an Arm project

A household folding robot doesn't carry a workstation — its brain is an Arm SoC (a Jetson-class board or smaller). watch2fold treats that as the design constraint from day one: the policy is sized for Arm-CPU inference, exported to portable ONNX, and re-verified on real Arm silicon on every push. The robotics is the demo; the deliverable is an optimization loop that ends on Arm.

What it does

One human folding demonstration becomes a deployed dual-arm policy with zero teleoperation:

1. Retarget the demo to dual SO-101 end-effector trajectories.
2. Adapt it across a situation grid (rotation ±15°, position ±4 cm, scale ±10%).
3. Judge & repair in simulation — a frozen geometric judge (area / IoU / corner / layers) approves every episode; failures get repaired and retried. Only approved episodes become training data.
4. Distill into a compact 21.4M-param ACT policy → ONNX.
5. Verify on Arm — GitHub's ubuntu-24.04-arm runner asserts x86↔aarch64 parity and measures latency on every push.

Optimization work & measured improvements

Inference, sized for Arm.

• 21.4M params (vs ~80M reference ACT), 76.4 MB fp32 ONNX, GroupNorm throughout so INT8 edge quantization stays on the table.
• Arm64 (aarch64 runner, CPU-only): 136.8 / 137.9 ms p50/p90 per 50-action chunk — 18× inside the 2500 ms chunk budget of the 20 Hz controller. Action chunking amortizes one inference over 50 control ticks, which is exactly what lets an Arm CPU drive a real-time arm without a GPU.
• Cross-architecture parity max|Δ| = 1.19e-06 (tol 1e-3): the artifact that passed on x86 is bit-faithful on Arm.

Data economics.

• Hundreds of teleop demos → 1 demo, 0 teleop minutes: the judge-gated loop approved 297/300 (99%) first-batch episodes, 350 total.
• One data-flywheel round (failure-targeted, diversity-augmented, same frozen judge) lifted closed-loop grid success 53.0% → 62.1% (66 rollouts) — the improvement mechanism is in-repo and repeatable.

Developer experience.

• Clone → build → generate → judge → test, top-to-bottom in the README with expected outputs at every step.
• The Arm verification path needs no simulator license: the CI workflow (or three commands on any Arm64 board) reproduces parity and latency numbers from the deployed ONNX alone.
• Robot integration is one framed-JSON TCP contract (reset/step/get_cloth) — the same protocol the evaluation grid drives today is the Jetson bridge tomorrow.

How we built it

C++17 RaiSim sim (deterministic generator + TCP server whose pin-grasp and assists are driven purely by the policy's own gripper targets), Python retarget/repair loop (MediaPipe → trajectories → repair ops), PyTorch CVAE-ACT with 3-camera observations, onnxruntime deployment. The cloth physics took 18 measured iterations — bend-compliance sweeps, force-capped pin grasping, app-layer friction anchors — before a fold looked and judged like a fold.

Challenges

XPBD cloth can't swing a human-style lift-over fold, so we converged on a surface spread-drag technique — and later found Sunday Robotics' Memo folds the same way in production. And we hold our evaluation to the same standard as our physics: two early measurement runs that turned out to be invalid (eval-harness defects we found ourselves) are archived and labeled in eval/, and every number above comes from the audited, reproducible run.

What's next

Jetson TensorRT FP16/INT8 over the same ONNX and TCP contract; cloth visual randomization; a crumple/flatten primitive; public bimanual datasets (HiFi-UMI-2K) as new teachers for the same loop.

Built With

• arm64
• c++
• hdf5
• mediapipe
• onnx
• opencv
• python
• pytorch
• raisim