Scrub

Inspiration

In an operating room, a scrub tech hands the surgeon each instrument and keeps count of everything that leaves the tray. Hospitals can't hire enough of them: a surgical tech job stays open about four months on average (Becker's Hospital Review). When nobody is there to pass, the surgeon looks away from the patient, puts down what they're holding, and searches the tray.

The count matters as much as the passing. A Mayo Clinic study found an item left inside a patient in about 1 of every 5,500 operations, and in most of those cases the count had been recorded as correct (Cima et al., 2008). Sharps matter too: the CDC estimates about 385,000 needle and sharps injuries a year among US hospital workers (CDC), which is why operating rooms pass scalpels hands-free.

What it does

Scrub is a robot arm that works as a second pair of hands.

Ask and receive. The surgeon says "hemostat" (or "Kelly", the name surgeons often use). Scrub picks it from the tray and brings it to the surgeon's open hand, handle first. It lets go when they pull, or when they say "got it."
Sharps never go hand to hand. Ask for the scalpel and Scrub sets it in a fixed safe spot and says "scalpel down."
Put-back. Set a used instrument in the return spot and Scrub puts it back in its slot on its own.
The count. Say "count" and Scrub reads back what's still out, checked against the camera. Say "check the tray" before a case and it says what's missing. Say "end of case" and a summary arrives by text.
Stop. "Stop", the spacebar or the stop button freezes the arm at once. A person has to reset it before it moves again.

The demo runs an operating room tray (scalpel handle, hemostat, Mayo scissors). A second tray, a robotics bench (needle-nose pliers, wire cutters, hex key), runs on the same model and the same code with new demonstrations and a new settings file.

How we built it

Voice. A phone on speaker calls Scrub's Photon number. ElevenLabs turns speech into text. Gemini turns the text into one action and one instrument from the current tray, using structured output, so it can't name an instrument that isn't there. "Stop" skips Gemini and goes straight to the arm.
The arm. An SO-101 arm runs Perceptron's Isaac 0.5 model, fine-tuned on demonstrations we recorded this weekend by moving a leader arm by hand. One model handles every instrument, picking and putting back, from a text instruction like "pick the hemostat". We recorded in bright, normal and dim light, because operating lights cause glare on steel.
Reaching the hand. An overhead camera runs MediaPipe to find the open hand. A camera-to-arm calibration turns the palm into a point the arm can reach, and the arm stops a few centimeters short.
Seeing the tray. Perceptron's vision API finds each instrument and maps it to its slot. That drives the grasp check, the automatic put-back, the count and the tray check.
The controller. One Python server owns the arm and every safety rule. Everything else sends it messages over a small fixed set of formats, which let four people build the parts at the same time.
The record. MongoDB Atlas logs every request, handoff time and test result. A dashboard shows the live camera, the current step, what's out and the success tables. A public page (on Vultr, at our GoDaddy domain) shows the count, the tables and a replay of each case.

Challenges we ran into

Thin, shiny instruments are hard to grip, and glare under bright light changes what the cameras see, so we recorded and tested every item in three lighting setups. We get only two training runs, so every recording is checked for frame rate and camera setup before it's used. The handoff also had to adapt to wherever the person's hand is, which is why reaching is planned from the hand tracker instead of learned.

Accomplishments that we're proud of

One system that hears a spoken request on a real phone call, picks the right instrument with a model we fine-tuned, hands it to a person, handles sharps safely, puts instruments back, and keeps an accurate count, with every step logged and shown live.

What we learned

How to collect demonstration data that a robot model can learn from: steady camera placement, one calibration, consistent motions, and variety in position and lighting. We also learned to keep the stop path separate from everything else, because a round trip to a language model is too slow when someone says stop.

What's next for Scrub

Surgery centers first, as a monthly subscription per operating room. Operating room time costs about $36 to $37 a minute (Childers and Maggard-Gibbons, JAMA Surgery, 2018), so saving a room even 20 minutes a month covers the fee. Scrub is a research prototype, so training labs come before live cases. Then a medical-grade arm, and handing over the next instrument before it's asked for, based on the step of the procedure.

Tracks

Community Kitchen: an assistant that fills a health care staffing gap and works for a judge live.
Perceptron Robotics: Isaac 0.5 fine-tuned on our own demonstrations, one model for every instrument, both directions, tested by lighting.
Perceptron Perception: their vision API runs the tray state, the grasp check, the put-back, the count and the tray check.
Photon (and best voice agent): the whole case runs over a call to our Photon number; texts and the end-of-case summary use the same thread.
DubHacks NEXT: surgery centers first, priced per operating room per month.
MLH Gemini API: speech to command with structured output, the end-of-case summary, and reading texts.
MLH MongoDB Atlas: every event, handoff time and test result; the success tables and case replays.
MLH Tiger Data: handoff times as a time series in a Tiger Data hypertable, shown on the public page.
MLH Backboard: Scrub's memory of what each clinician asks for and what usually comes next.
MLH Vultr: the public judge page runs on a Vultr server.
MLH GoDaddy Registry: our domain points at the public page.
MLH Commit Fellowship.
MLH DEV writing: a post about building Scrub after the event.