Second Look

Inspiration

In a gallbladder removal, the surgeon must prove to themselves, before clipping anything, that they are looking at the cystic duct and artery and not the common bile duct. The checklist for that is the Critical View of Safety (CVS): two structures isolated, the hepatocystic triangle cleared, the cystic plate exposed. Getting it wrong is rare and severe. Bile duct injury occurs in 0.32 to 0.52% of cases (505,292 patients, Pucher, Surg Endosc 2018). In a study of 1,570,361 Medicare cholecystectomies, 7,911 patients had a bile duct injury, and their adjusted hazard of death was 2.79 times higher (95% CI 2.71 to 2.88). At 9.2 years, 19.5% of injured patients were alive, against 55.2% of uninjured patients (Flum, JAMA 2003).

The cause is perception, not skill with the instruments. In a review of 252 injuries, 97% began with a visual misperception, and only 25% were recognised during the operation (Way, Ann Surg 2003). Surgeons also believe they reached the safe view when they did not. In a US study, 8 of 10 practising surgeons did not achieve CVS on video, and two of them had dictated that they had (Stefanidis, Surg Endosc 2017). In the Netherlands, operative notes said CVS was reached in 80% of complicated cases, while video review found 10.8%, and none of the patients with a biliary injury had it (Nijssen, World J Surg 2015).

That gap ends up in court. In Vanderhoof v. Berk (2015 IL App (1st) 132927) the patient died of septic shock after the surgeon "severed the patient's common bile duct in error after he misidentified it as the cystic duct." The trial turned on the critical view of safety, and the jury awarded $910,742.79, affirmed on appeal. In Back v. Facey (2026 NY Slip Op 03858) the surgeon testified he took "all of the necessary steps" to establish the critical view; imaging later showed clips blocking the common bile duct, and the patient needed an open reconstruction. That case is still going. I read the full opinions for both. In the UK, 418 NHS claims followed laparoscopic cholecystectomy between 1995 and 2009, with £20.4M paid, and bile duct injury was the most common claim and the most likely to succeed (Alkhaffaf, Ann Surg 2010). Juries do not always side with plaintiffs, and I make no claim about outcomes in general. The shared fact pattern is a dispute over whether CVS was actually achieved, and nobody has an independent, time-stamped record to settle it.

What it does

Second Look is a clip gate for laparoscopic cholecystectomy. It reads key frames from a real operation and answers one question: was the critical view of safety established before the clip?

For each of six real cases the app shows:

• A Clip Gate card with the three criteria, the surgeon raters' count beside Second Look's call, and a verdict: "Clear to clip" or "Hold", with the missing criterion named (for example "Hold. Cystic plate not exposed.").
• A 12-frame verification ledger, one row per key frame, five seconds apart over the last minute. Each row is marked silent, log or speak, with three bars for the criteria.
• An opened row that shows the frame itself, each criterion's call, the two vision models' scores out of 10, the calibrated chance the criterion is met, and the surgeons' count (for example 3 of 3).
• A Run it button that re-reads a case's last frame live (one vision call per eye model, one Nemotron call, about 13 seconds).

The default output is silence. The agent speaks only when an instrument is at a duct or the sequence has reached the moment before the first clip, and the criteria are not all established.

How I built it

The backend is FastAPI. The UI is a light console in plain HTML, CSS and JavaScript with no build step. Everything goes through Nebius Token Factory, which is OpenAI-compatible, so one base URL and one client covered every model.

No NVIDIA model reads images on serverless Token Factory, so the pipeline splits perception from decision:

1. Two vision models are the eyes. Qwen/Qwen3.8-27B and zai-org/GLM-5.3-Flash each score the three questions 0 to 10 per frame, against eight labelled reference frames (Endoscapes vid168 and vid190, including boundary cases where only 2 of 3 surgeons said yes). They describe; they do not decide.
2. A median filter over a trailing three-frame window smooths the scores.
3. Calibration. One small logistic regression per criterion, over the two models' windowed scores, turns them into a chance of "met". They are fitted on 47 other labelled frames (vid168 and vid190 only), with classes weighted equally because those videos have more cystic-plate-positive frames than the test videos.
4. Nemotron Super (nvidia/nemotron-3-super-120b-a12b) reads the calibrated yes/no per frame and decides every criterion state, the silent, log or speak action, and the gate.
5. Nemotron Ultra (nvidia/Nemotron-3-Ultra-550b-a55b) audits the gate's log against the rule once per case and writes the debrief.

The 72 test frames and 47 training frames are read ahead of time by both eye models and cached, so visitors cost no tokens. I tried Kimi-K2.6 as a third eye and dropped it: about 70 seconds a call, repeated timeouts, and only 48 of 119 frames read.

The footage is real: SAGES CVS Challenge 2024 clips (CC BY-NC 4.0) and Endoscapes2023 frames (CC BY-NC-SA 4.0), each shown beside its surgeon labels. The Endoscapes cases are 4x time-lapse of one frame per second, and the app says so.

What makes it agentic

Nemotron makes every decision in the right-hand column of that table; the vision models only describe. After the case, Nemotron Ultra checks the whole log against the rule, so the same system that decided is audited by a stronger model.

Accomplishments

I wanted a number I could defend, so I held out data and measured.

• Test set. The last 60 seconds of the six videos the app shows (12 labelled frames, 5 seconds apart), judged on three-frame windows: 10 windows per video, 60 in all, 180 criterion judgements, scored against the surgeon majority (2 of 3).
• 90.0% correct on held-out judgements (162 of 180), against 64.4% (116 of 180) for always answering "not met".
• All three criteria now beat that baseline: two structures 56 of 60 (always-no: 37), triangle 51 (33), cystic plate 55 (46). The cystic plate is the hardest, and the published purpose-built models I found sit around 71% balanced accuracy (DeepCVS) or 0.73 on that criterion (SurgSmart 2026); the numbers are not directly comparable.
• "Clear" precision of 0.89. Of 14 windows the surgeons truly rated all-three-met, Second Look said clear on 9: 8 correct and 1 wrong (sensitivity 0.57). The one false clear was a borderline window in clip 144fef8d.
• The gate matches the surgeons on all six cases: clear on e60de21a and vid194, hold on 144fef8d (cystic plate not exposed, which the surgeons agree on), 280aaeb3, 0041dfb1 and vid163.
• Nemotron's per-frame states reproduce the calibrated check exactly (162 of 180 either way).

I would rather be straight about what that proves. It is six videos, and frames within a video are not independent, so 162 of 180 is a measured number on a small sample, not a guarantee. I compared several choices on the test set (a single cut-off per criterion, then the regression, then equal class weights after the first fit over-predicted "met"); other variants scored 76% to 85%, and the regression over both models was best at 90%, so expect the real figure to be lower. Only two of the six videos genuinely clear at the end, so a clear verdict has been checked on those two and on four holds. For the SAGES clips the decision moment is the end of the 90 s clip, not a detected clipping event. The tools/fit.py and tools/evaluate.py scripts reproduce every variant and number.

What's next

• Score the full SAGES test set (300 clips) as a batch and publish agreement against the raters, misses included.
• Detect the clip applier directly, so the decision moment is a real event and not the end of a clip or sequence.
• Run on full-motion footage with CVS labels. Endoscapes videos 1, 2, 3, 4, 7 and 121 are the same operations as Cholec80 videos 67, 68, 70, 71, 72 and 66, so full-motion versions of labelled cases exist.
• Add the physical layer the Physical AI track is about: a camera pointed at the operating-room monitor, a spoken warning, and a green, amber or red light on the tower. This is a design, not something I have built or tested.
• Write the one-page CVS record for the chart: the best frame per criterion with time stamps, and what was never seen.

Built With

• fastapi
• nebius-token-factory
• nvidia-nemotron-3-super&ultra
• python
• qwen3.8-27b