SirGen

Inspiration

Surgeons already have a lot of screens. What they don't have is a teammate who is watching the whole case. the camera, the patient's vitals, the chart, and where the operation is in the procedure. And a teammate is only useful if they know when to stay quiet. Nobody wants a voice in their ear while they are placing a clip.

We wanted to build that teammate, and make it run on one box in the room.

What it does

SirGen follows a laparoscopic cholecystectomy (gallbladder removal) live. It watches the video, tracks the phase of the procedure, and reads the vitals and the patient chart together.

8 agents that disagree in the open. A surgical agent (video), an anesthesia agent (vitals and drugs) and a record agent (chart) each give a view. When a blood pressure drop shows up, they work out whether it looks like a real bleed, a drug effect, or a sensor artifact. Plain code decides when they have converged, not another model.
It knows when to speak. If the surgeon is in a critical step, the assistant holds non-urgent messages and releases them when the moment is safe. A real emergency still breaks through.
Minimal surgeon screen. Faint, almost transparent, only vitals, status and potential issues.
A tamper-evident audit log. Every event goes into a hash-chained record, and the post-op report is drafted from that log. The aftercare and medication sections are a draft for clinician review that only cites the chart and the log.

How we built it

Everything runs on an ASUS Ascent GX10. Nemotron Lightning (NVFP4) is served with vLLM and powers the agents. A surgical-tool detector we trained on Cholec80 reaches 0.82 mAP50 on a held-out video and runs in about 2 ms per frame. The backend is Python and FastAPI, the interface is React and TypeScript, and the pitch slides use three.js.

The agents only read what the system sees live: video output, vitals, drugs and the chart. We do not feed them scripted evidence, and we score them against a fixed blood-pressure-rule baseline on three cases: true bleed, drug effect and artifact.

Challenges

Real surgical footage with real vitals from the same patient does not exist in public data. Our demo is a labelled composite of two de-identified public cases (Cholec80 video and VitalDB case 59), plus a synthetic chart.
Our only real clip has no bleeding in it, so the bleed case is the hardest to show honestly. We lead with the drug effect and artifact cases, where the agents reason on real vitals.
Keeping an assistant quiet is harder than making it talk. Most of the work was deciding when not to speak.
Sharing one GPU between the model, the vision models and training caused timeouts until we separated the jobs.

What we learned

The interesting part of a clinical assistant is not the answer, it is the timing and the evidence. We learned to make the agents show their reasoning in a form a person can read in two seconds, and to let plain code decide the final call.