Inspiration

In open-heart surgery, the surgeon's hands never leave the field. Every instrument they need is called out, often in shorthand like "needle holder, 6-0 Prolene, backhand", and a scrub tech has to find it among dozens of look-alikes, pass it handle-first, and count it back in at the end. Leaving an instrument inside a patient should be a "never event," yet The Joint Commission logged 119 retained-foreign-object sentinel events in 2024, the most in five years, and X-rays catch only about two-thirds of retained items. Meanwhile the US needs roughly 8,600 new surgical technologists and assistants every year while accredited training programs have fallen from 500+ to about 430. One of us grew up around cardiac surgery, and talking it through with a practicing heart surgeon convinced us the fetch-pass-count loop is exactly the kind of repetitive, high-stakes work a robot should take off a person's plate.

What it does

Surgio is a robotic scrub tech. You say " twenty blade." Surgio:

Hears the request (Cartesia Ink speech-to-text) and turns it into an intent with Claude. Every instrument Claude names must pass the arm's own tray catalog, so a hallucinated tool gets rejected.
Sees the table through a wrist camera and identifies surgical instruments by measured shape.
Picks the right one with a soft stall-then-squeeze grasp, checks on camera that it actually lifted it, and retries up to 3 times.
Hands it over: it presents the tool and releases only when the camera sees your hand take it.
Counts: every instrument is tracked tray → surgeon → used on a live Convex board, and the end-of-case count names anything missing, out loud.

Every hand-off is recorded (faces blurred on-device) and labelled as a training episode.

How we built it

Hardware. A Seeed reBot B601-RS arm: 6 joints + gripper on RobStride motors over a CAN bus (PCAN-USB, 1 Mbps), a USB wrist camera mounted on the gripper, and 3D-printed red replicas of real cardiac instruments (#10/#20/long scalpels, fine vascular forceps, forward/back micro scissors, Ryder and short needle holders).

Arm control (arm/). A 200 Hz control loop with minimum-jerk moves, a collision guard (tracking error > 25° freezes the arm), and a torque-capped gripper ramp. Kinematics come from the URDF (numpy FK/Jacobian/DLS IK, plus Pinocchio). The gripper points straight down when

$$q_2 - q_3 - q_4 = 90^\circ$$

since joints 2–4 pitch about parallel axes. Picks hover 10 cm above the grab point at the scan pose's 45° tilt, then make one straight move down to

$$z_{\text{grasp}} = z_{\text{table}} + d_{\text{tilt}} + \min!\left(\tfrac{t}{2},\ 1\,\text{mm}\right)$$

where $ t $ is the tool's thickness and $ d_{\text{tilt}} = 7 $ mm is the measured fingertip drop when tilted. The jaws close until they stall on the tool, then 4° more for grip, and never past the calibrated closed limit.

Vision (perception/). Red pixels (Lab $ a^* > 28 $, grown into $ a^* > 16 $) are projected through forward kinematics, a ChArUco intrinsic calibration (0.21 px RMS) and a hand-eye calibration into a metric top-down map of the table, so every part is measured in millimetres no matter where the camera is. Each blob gets six features $ f $ (length, mean/median width, narrow-end width, taper, solidity) and the nearest class scaled by that class's spread:

$$c^* = \arg\min_{c} \sum_{k=1}^{6} \left( \frac{f_k - \mu_{c,k}}{\sigma_{c,k}} \right)^2$$

Fitted on ~150 real wrist-camera views (600+ part samples), it got 240/241 parts right on a held-out layout. Grab points are the box centre for scalpels and forceps and the thinnest cross-section for scissors and needle holders.

Hand-off. Sensing the tug through the joints moved them less than 0.1°, so the camera watches instead: the tool counts as taken when its red pixels near the jaws stay below 30% of the held count for 0.5 s.

Console (console/). Cartesia Ink and Sonic for speech in and out (with a pronunciation dictionary for words like Castroviejo and Vicryl); Claude with strict tool use and prompt caching for intent; voice, arm and "eyes" as Fetch.ai uAgents, plus an ASI:One chat agent on Agentverse; Convex for the live instrument board, events, 20 Hz arm telemetry and audit-log anchors; Bitdeer Model Studio for the end-of-case report; Factory Droid for human-approved fault triage. The arm's HTTP API is locked down with a bearer token, Origin/Host checks and a hash-chained, HMAC-signed audit log whose head is mirrored to Convex, so cutting the end off the log raises an alert.

Simulation (sim/). A MuJoCo digital twin where a camera-guided residual SAC policy (visual servoing + a learned residual) reaches targets 99–100% of the time — our path to picking out of cluttered piles.

Challenges we ran into

RL Training Loop Compatibility Issues Training a kinematics/inverse-kinematics model with our closed RL training loop required way too much data and more training time than we had at the hackathon. It was highly accurate but sometimes imprecise.
Colour tracking failed on dark instruments. A dark clamp looked identical to the black jaws, cables and bags. We moved to bright red prints, masked out the fingers, and measured shape instead of trusting colour.
Sim-to-real. Our YOLO detector hit mAP50 0.995 on rendered data and didn't transfer to real photos. The hand-built metric classifier was more honest and more accurate.
We broke the gripper by commanding it past its closed limit. Now it can never be commanded past closed_deg.
The camera got knocked mid-hackathon and every pick landed ~20 mm off. We built grasp_calib.py to re-measure the grab offset on bare paper in minutes.

Accomplishments that we're proud of

A real arm that hears a request, picks the right instrument on the first try, and hands it to a person, releasing only when it sees the tool taken.
240/241 held-out identifications from a classifier we built and calibrated in a weekend.
A system that fails closed: unknown, ambiguous or refused inputs produce a question, never a motion.
99–100% camera-guided reach success in simulation.

Challenges we ran into

Colour tracking failed on dark instruments. A dark clamp looked identical to the black jaws, cables and bags. We moved to bright red prints, masked out the fingers, and measured shape instead of trusting colour.
Sim-to-real. Our YOLO detector hit mAP50 0.995 on rendered data and didn't transfer to real photos. The hand-built metric classifier was more honest and more accurate.
We broke the gripper by commanding it past its closed limit. Now it can never be commanded past closed_deg.
The camera got knocked mid-hackathon and every pick landed ~20 mm off. We built grasp_calib.py to re-measure the grab offset on bare paper in minutes.
Speech-to-text vs. surgical vocabulary. "Resano forceps" came back as "Rosano Forceps," and our matcher let the generic word "forceps" win, silently fetching the wrong tool. Named instruments now always beat category words, and unknown ones are refused.
Plumbing: RobStride only accepts a mode change while disabled; only one process can open the CAN adapter on macOS; and a Python client that held the GIL for each ~110 ms cloud round trip froze our 20 Hz telemetry until we switched to Convex's HTTP API.

What we learned

Classic, measured computer vision beat a model trained on renders when the real camera was the judge. Calibration isn't a one-time step; it drifts and needs a fast re-measure tool. And for anything near a patient, the most important feature is what the system refuses to do.

Accomplishments that we're proud of

A real arm that hears a request, picks the right instrument on the first try, and hands it to a person, releasing only when it sees the tool taken.
240/241 held-out identifications from a classifier we built and calibrated in a weekend.
A system that fails closed: unknown, ambiguous or refused inputs produce a question, never a motion.
99–100% camera-guided reach success in simulation.

What's next for Surgio

Real stainless instruments on a crowded tray, with a detector trained on real photos.
Piles, not layouts: our RL reach on the real arm, plus decluttering.
Clinical safety: hardware e-stop, verified force limits, sterile draping, and a study with working scrub techs before any claim of benefit.