Inspiration

On any given day, about 1 in 38 patients in U.S. hospitals has a healthcare associated infection (CDC, 2026). In low and middle income countries, 11 percent of surgical patients are infected during surgery. Many infections start with a small slip, like a gloved hand brushing a non sterile table and then touching an instrument.

Instructors teach sterile technique by watching mock operations, but one person cannot follow every hand in the room at once, and a slip nobody sees never gets corrected. We wanted to give instructors a second look: a tool that watches every hand, marks the moments worth checking, and leaves the final call to the instructor.

What it does

reScrub reviews a recorded practice procedure from one camera.

Labels everyone anonymously as Person 1, Person 2, and so on
Tracks arms, hands and fingertips
Marks sterile and non sterile surfaces
Flags timestamped moments where a break in technique may have happened, including chains like glove → side table → forceps

The instructor clicks a finding, the video jumps to that moment, and they confirm or dismiss it. When the review is done, Gemini writes a short debrief from the confirmed findings that can be printed or exported. reScrub suggests and a human decides.

How we built it

Website: Next.js, TypeScript and Tailwind. The annotated video, timeline and findings list stay in sync, so clicking any one moves the other two.

API: FastAPI in Docker, with Argon2 password hashing and secure session cookies.

Analysis pipeline (GPU worker):

People: YOLO26 pose and BoT SORT track everyone. An appearance model plus our own merging rules keep Person 1 as Person 1 when the tracker loses them.
Hands: RTMW finds 21 points on each hand.
Surfaces: a segmentation model we trained on the MM OR surgical dataset finds drapes, instrument tables and other surfaces and marks each as sterile or non sterile.
Depth: Depth Anything V2 helps tell hovering from touching.
Contacts: a fingertip counts as touching a surface when it is at or inside the surface edge in the image, at a similar depth, and moving slowly enough to be a real touch:

$$ \text{contact} \iff d_{\text{px}} \le 8 \;\wedge\; \lvert z_{\text{hand}} - z_{\text{surface}} \rvert \le 0.08 \;\wedge\; v \le 400\ \text{px/s} $$

where $ d_{\text{px}} $ is the distance to the surface edge in pixels, $ z $ is relative depth from 0 to 1, and $ v $ is fingertip speed.

Findings: rules turn contacts into timestamped findings and contact chains, and the annotated video is rendered.

Tiger Cloud (TimescaleDB): every detection, keypoint and contact is stored as time series data and compressed after a week. pgvector stores each person's appearance so their label stays stable. The database is also the job queue between the website and the GPU worker, so the two servers never talk to each other directly.

Gemini: writes the final report from the findings and the instructor's decisions. It never sees the video, and we check every answer so it can only mention findings that really exist.

Vultr: one server hosts the website and API. A second server with an NVIDIA A16 GPU and 200 GB of block storage runs analysis and training. Videos live in private Object Storage. The site is live at rescrub.health.

Challenges we ran into

Hovering looks like touching. From one camera, a hand in front of a surface can look like it is on it. Depth helps, but unclear moments stay marked as uncertain instead of being counted as breaches.
Everyone looks the same. Identical gowns, caps and masks made normal tracking swap people. We added appearance matching with position checks so a returning person gets their old label back.
Sterility is invisible. Labels come from trained surface types, not guesses about color or texture.
Huge, slow data. The full dataset is 432 GB, so we built smaller downloads of a few surgeries and trained on our GPU server. MM OR records one frame per second, so the pipeline had to keep exact timestamps on choppy video.
Honest testing. We split data by surgery, not by frame. Otherwise near identical frames from one operation land in both training and testing and the model looks better than it is. Our surface model scores

$$ \text{mask mAP}_{50} \approx 0.65 $$

on a surgery it never saw. A longer training run stopped improving after about 13 epochs, so we kept the smaller model.

Deployment. We hit database certificate checks failing inside Docker, conflicting Docker packages, a training job that ran on the CPU by mistake, and Gemini answers getting cut off because the newest model's reasoning used up the output limit. Each one taught us to read the error before changing anything.
Keeping secrets safe. Passwords and API keys live only in locked down files on the servers, entered at hidden prompts and never committed to Git.

Accomplishments that we're proud of

We built a complete, live system that goes from upload to anonymous tracking to timestamped findings to instructor review to a written debrief. A three minute clip takes about a minute to analyze. We built it responsibly, with private storage, anonymous labels, honest uncertainty, measured accuracy, and a human making every final call.

What we learned

How to connect a real system: a website, an API, a time series database, cloud storage, GPU workers and an AI model, each failing safely when another is down.
Computer vision in real rooms: crowded scenes, matching gowns and a single camera break assumptions that work on clean demo videos.
Testing matters as much as the model: how you split data can make a weak model look strong, and a lower score you can trust is worth more than a high one you can't.
Using AI carefully: Gemini writes a clear report fast, but only when we limit it to verified facts and check what it returns.
Teamwork: Git, pull requests and code review, and deploying changes without breaking the live site.

What's next for reScrub

Test with instructors at simulation labs and measure how often our findings match theirs
Add more camera angles so fewer hands are hidden
Finish training our own identity model
Show how a trainee improves across sessions