-
-
Replay pair B3: Llama and Qwen pass a rewrite that lost its diagnosis; Nemotron Nano flags it. Open it: Replay, Part B, pair B3.
-
Replay, Part A, case A1: one text end to end (input, rewrite, three verdicts). One of 12 recorded runs, not a representative sample.
-
Teacher, student, gate and panel selector, with the Nebius service that runs each. Super wrote 7,983 references; Nano is the third judge.
-
Nemotron Nano catches 110 of 120 hand-verified dropped diagnoses vs 56 each for Llama and Qwen, but also flags 53 of 120 clean controls.
-
Adding Nano shows no detectable change in judge agreement, but Phi_V, scored against hand-verified errors, rises from 0.476 to 0.553.
-
gpt-oss-120b and DeepSeek-V4-Flash lead, statistically tied. Within Nemotron, scale buys no measurable gain: 30B Nano vs 550B Ultra.
-
Replay, Part B, pair B3: the recorded and committed verdicts agree. Llama and Qwen say SAFE, Nemotron Nano says UNSAFE; the gate flags it.
-
45 of the gate's 99 DISAGREE flags were false alarms at 50% drops. Research prototype, not a medical device, not for real patient data.
What if the AI quietly drops a diagnosis? MediSimplifier rewrites hospital discharge summaries into plain language. Three LLM judges check each rewrite against the original, and the gate flags rewrites that may have lost a diagnosis.
Try it in 30 seconds (no account, no GPU): open the demo, which opens on Replay. Part A shows the product end to end on 12 recorded texts: the input, the plain-language rewrite and the three judges' verdicts. Then open Part B, pair B3, a breast-tumor summary whose rewrite no longer names the granular cell tumor. Llama and Qwen say SAFE, NVIDIA Nemotron Nano says UNSAFE, so the gate flags it.
1. The finding, and its price. On 120 benchmark rewrites with the primary diagnosis removed (the author verified each removal by hand), Llama-3.3-70B and Qwen3-32B each catch only 56 under the deployed gate prompt. NVIDIA Nemotron Nano catches 110 (92%), and it also flags 53 of the 120 clean controls (44%).
2. Agreement is not validity. Adding Nano does not detectably change how much the judges agree (Fleiss κ 0.214 → 0.179, and the confidence interval includes zero), so an agreement metric would have missed the fix. Our framework, VAGT (Veridicality-Anchored G-Theory), scores a panel's dependability, Φ_V, against hand-verified errors instead: it rises from 0.476 to 0.553 (paired Δ +0.0765, 95% CI [+0.0516, +0.0992]) at the benchmark's 50% drop share. At 20% drops the lift is not detectable.
3. A tool that decides. /v1/audit_panel recommends which judge to add to a panel, with a bootstrap CI, from a pre-computed pool of eight judges, and it answers live in the demo. On our data it picks gpt-oss-120b (statistically tied with DeepSeek-V4-Flash; we added the tie-break rule after seeing the result). The gate still runs Nano because gpt-oss-120b was also the benchmark's oracle, so its lead is partly by construction; Nano's figures are free of that effect. A 30B Nemotron Nano is statistically indistinguishable from the 550B Nemotron Ultra (ΔΦ_V +0.0765 vs +0.0781).
4. Limits, stated up front. The gate raises a flag, not a verdict: in a dual-auditor review of flagged outputs, both auditors confirmed a genuine diagnosis drop in 2 of 20 cases, and 6 of 20 are contested. The blind spot appears under the deployed gate prompt only: under the calibration prompt Llama catches 114 of the 120 drops, though it flags 98 of the 120 clean controls. An independent physician marked Unsafe all 8 rewrites with the primary diagnosis removed, 4 of which the gate had passed; the 16 pairs reviewed were chosen by design (counts, not rates). Our first, automated benchmark had about 85% mislabeled diagnosis-drop labels; we rebuilt it, the recommendation changed from Nano to gpt-oss-120b, and we left the superseded +0.071 lift visible in the README instead of scrubbing it. This is a research prototype, not a medical device.
Inspiration
Patients go home with discharge summaries written for clinicians, dense with abbreviations and diagnoses most people cannot read. An LLM can rewrite them in plain language, but the worst failure is the quiet one: a fluent rewrite that no longer mentions a diagnosis.
In our v1 project (first place in the Nebius Serverless AI Builders Challenge), two LLM judges, Llama-3.3-70B and Qwen3-32B, checked 1,001 rewrites and barely agreed on whether they were medically faithful (Cohen's κ = 0.11; Llama flagged 67.5% as unsafe, Qwen 11.2%). That raised the question behind this project: when judges agree, or disagree, does it tell you whether the panel is right? Agreement measures consensus, not truth. We developed VAGT between the two submissions, and v2 applies it to real judge verdicts.
What it does
MediSimplifier is two products on one safety layer.
- Safe simplification.
POST /v1/simplifyreturns a plain-language rewrite at about a 9th-grade reading level plus the safety gate's verdict. Qwen3-32B and Nemotron Nano decide, Llama-3.3-70B is advisory, andsafety_modecan beflag,block, orstrict(which also withholds when the deciding judges disagree). The gate also runs on its own, to check any simplifier's output. - Judge-panel selection.
POST /v1/audit_panelrecommends which judge to add to a panel to fix its blind spot, with the expected gain inΦ_Vand a bootstrap confidence interval.
The demo needs no account and no GPU: Replay shows 28 recorded cases (12 end-to-end runs of the product, each a rewrite with the gate's verdict, and 16 gate cases), and its panel section runs /v1/audit_panel live. Live sends your own text to a Safe Endpoint. Ours runs on three endpoints that together cost about $14/h, so they are stopped between demos. To try Live at no cost to you, write to [[email protected]] with the time you plan to look and I will start them for your session (they take about 8 to 10 minutes to load), or deploy your own.
**Please note: Nebius has announced that it ends API support for the serverless Nemotron Nano model on 9 November 2026 (we were told by email on 30 October). The gate calls that model as one of its two deciding judges, so after that date a Live session shows ERROR for the Nemotron judge, the gate's verdict is ERROR unless Qwen judges the rewrite unsafe, and a flagged rewrite has no explanation. The Qwen and Llama judges keep working. Replay shows all three judges on 28 recorded cases, and the panel makes no model calls, so both are unaffected. Details: README, B8, item 5.
How we built it
- Teacher. NVIDIA Nemotron Super (120B) on Nebius Token Factory wrote our training references (7,983 in the training set) from synthetic clinical notes with no real patient data: 9,999 calls, 83.90M output tokens, USD 75.51, no idle cost.
- Student. OpenBioLLM-8B, fine-tuned with LoRA (Nebius Jobs on H100), merged, published on Hugging Face, and served with vLLM on a Nebius GPU Endpoint.
- Gate. Nano runs per-token on Token Factory; Qwen3-32B and Llama-3.3-70B run on Token Factory dedicated endpoints.
- VAGT. A hand-verified benchmark of 120 primary-diagnosis drops and 120 paired clean controls. Six candidate judges (Nemotron Nano, Super and Ultra, gpt-oss-120b, DeepSeek-V4-Flash, gemma-3-27b-it) were each added to the Llama and Qwen incumbents, and
/v1/audit_panelserves the results from a Nebius CPU endpoint.
Challenges we ran into
- Our first benchmark was wrong. A hand audit of 150 automated "dropped diagnosis" items found that 128 still contained the diagnosis. We rebuilt the benchmark from hand-verified drops, and that changed the recommendation.
- The price of catching drops. Part of Nano's 44% false-flag rate is paraphrase mismatch: "fast-growing blood cancer" for "leukemia" gets flagged as a dropped term. The deployed rule accepts false alarms in exchange for recall, on purpose.
- Judges move. Qwen3-32B left Token Factory's serverless catalog during development, and serverless Llama-3.3-70B returns 403 on our account, so both run on dedicated endpoints. Re-running Qwen on the same 240 items changed its verdict on 41 of them.
Accomplishments that we're proud of
- Controls showing that the lift is detection, not a higher flag rate: four of the six candidate judges beat all 1,000 of their own shuffled-verdict nulls (p = 0.001).
- A decision tool you can call, kept running through judging, with committed receipts of what it returned.
- An open, reproducible package: the merged model, both versions of the benchmark and the training dataset are public on Hugging Face, with deployment steps in the repository.
What we learned
- Agreement is not validity. Llama and Qwen pass the same 41 of the 120 drops: they agree, and they are both wrong.
- Scale was not what mattered. Within the Nemotron family, 30B to 550B bought no measurable gain; what mattered was whether a model could catch the error. gpt-oss-120b and DeepSeek-V4-Flash led the table, and gemma-3-27b-it did not move the blind spot (
ΔΦ_V+0.0066). - Clean labels can flip a headline. Our first pass recommended Nano; on hand-verified labels it recommended gpt-oss-120b.
- A catch rate needs its false-alarm rate beside it.
- Field notes for Nemotron users. Super returns an empty answer with no error when a 1,024-token budget goes to reasoning (we needed about 16,000). Nano's verdict has to be read out of up to 8,000 tokens of reasoning text, so there is no score to put a threshold on. A pre-registered test found no demonstrated gain from a narrower faithfulness prompt. Five findings, with their scripts and numbers: docs/NEMOTRON_INSIGHTS.md.
What's next for MediSimplifier — Nemotron × VAGT
- Physician-labeled validation of the gate beyond the 16 pairs we have.
- Semantic grounding (medical NLI or physician judgment), so a correct plain phrase is not flagged as a dropped term.
- Measure the served model's real drop rate and report
Φ_Vat it, since the lift is not detectable at 20% drops. - Hand-verify the dose, negation and lateral strata, add a check for added content, score new judges live in
/v1/audit_panel, and replicate VAGT on a second benchmark or domain.
