Han dynasty

Past in the days, to control a robotic arm requires expensive tools. Instead, we ehought about what is we can use commercial deices to do it, that is Han dynasty, we used Apple watch as the interface

Comment

Inspiration

Robot arms are getting cheap. The SO-101 costs a few hundred dollars, and vision-language-action (VLA) models can now do real manipulation. Controlling them is still clunky, though: you need a leader arm, a gamepad, or a laptop. We wanted to control a robot arm with something already on your wrist. Point with your hand to steer it. Tap your fingers to take over. Say what you want, and let a VLA model finish the job. We also wanted it to be safe enough that a person could stand next to it.

What it does

Han Dynasty turns an Apple Watch into the controller for an SO-101 robot arm:

  • Double Tap (the watch's finger-pinch gesture) takes control. The watch calibrates to your wrist pose and puts the arm in MANUAL mode.
  • In MANUAL, your wrist motion drives the arm: tilt to lift, rotate to pan.
  • Double Tap again and speak a command, for example "close" or "pick up the cup". The watch records it, the iPhone transcribes it on-device, and the robot agent matches it to a known task. The plan is to hand that task to a VLA policy.
  • STOP is always one tap away. Every tap vibrates, so you can tell by feel whether the robot accepted the command or ignored it.
  • If the watch link drops or lags, the arm freezes in place within 250 ms. If it stays down, the arm goes to a safe HOLD.

How we built it

  • Apple Watch app (SwiftUI, watchOS 27): streams Core Motion data and handles Double Tap, voice recording, and haptics. An HKWorkoutSession keeps it running when the screen dims.
  • iPhone relay app: bridges WatchConnectivity to a pinned-TLS WebSocket to the robot. It also transcribes voice on-device with SFSpeechRecognizer.
  • Local Robot Agent (Python, asyncio):
    • a supervisor state machine (HOLD â CALIBRATING â MANUAL â LISTENING â VLA)
    • control epochs, so a stale command can never act on a newer state
    • a safety gate with soft limits, per-tick step limits, and a two-tier motion watchdog
    • a haptic policy and a task router
  • Hardware: SO-101 follower arm driven through LeRobot's Feetech bus, running in Docker on an AMD gfx1151 box. Two USB cameras (wrist and table) are wired up for the VLA.
  • Testing: 145 Python tests, including acceptance tests TC01âTC12 and randomized FSM invariant tests. Swift unit tests, plus an end-to-end UI test that drives watch â iPhone â gateway in simulators.
  • Shared protocol: messages are defined once and checked against fixtures in both Python and Swift, so both sides always agree on the format.

Challenges we ran into

  • The watch falls asleep. When you lower your wrist, the screen dims and watchOS cuts the link to the phone. Extended runtime sessions didn't help. We switched to a workout session, then found that watchOS can report the phone as "unreachable" even when messages still get through.
  • Double Tap kept going missing. We traced it to three separate causes:
    • AssistiveTouch takes over the gesture.
    • A disabled button silently swallows it.
    • Double Tap only fires for a button that is on screen, so scrolling down to the debug log killed it.
  • Campus Wi-Fi. Client isolation, blocked Bonjour, a Mac and watch that wouldn't pair, and changing IPs.
  • iOS TLS rules. Our pinned self-signed certificate passed our own check, but iOS still rejected it: first over a SAN mismatch, then under App Transport Security on a public IP. One documented quirk: setting NSAllowsLocalNetworking makes iOS ignore NSAllowsArbitraryLoads. We found all of this by pulling the iPhone's system log over USB.
  • Real hardware safety. Our probe found the arm resting outside its own servo limits. Turning torque on there would have made it jump, so we added a check that refuses to start in that case.

Accomplishments that we're proud of

  - The full watch → phone → robot agent loop works on real devices: Double Tap → calibrate → MANUAL, and spoken commands transcribed and delivered to the agent.
  - The real SO-101 is connected with torque on and holding position, and every safety check passed. Nothing moved unexpectedly.
  - The safety design holds up under failure. The arm freezes on lag, holds when the link is lost, and a stale or duplicate command can never move it.
  - Every interaction gives feedback by feel. You always know whether the robot heard you without looking at a screen.
  - On-screen debug panels on both the watch and the phone made debugging on real devices quick.

  What we learned

  - On wearables, the hard part is keeping the app running and connected, not the control algorithm.
  - Safety has to be designed in from the start: epochs, watchdogs, refusing to act when unsure, and never letting the arm drop.
  - Logs on the device itself beat guessing. Many of our "mystery" bugs turned out to have a one-line cause once we could see the real error.
  - Test against real hardware early. Simulators never showed us the TLS, ATS, Double Tap, or screen-dimming problems.

  What's next for Han Dynasty

  - Replace the placeholder joint mapping with real inverse kinematics, so the gripper tracks your hand in 3D.
  - Gripper control with the Digital Crown, plus grip detection based on motor current.
  - VLA hand-off: connect the voice-selected task to a SmolVLA policy using the wrist and table cameras. When it finishes, the arm returns to HOLD and waits for your Double Tap.
  - Raise wrist streaming from about 15–20 Hz to 50 Hz for smoother control.
  - Record demonstrations straight from watch teleoperation to train new VLA tasks.

Built With

Share this project:

Updates

Submission history