CodeSentinel cuts a risk-adjustment coder's review queue 39.4% at 1.000 precision on hard negatives, and sends only the charts that need it to Super and Ultra. On that 66-chart suite it costs \$0.0014 per chart on Nebius Token Factory, which works out to
$$ \frac{1 \text{ USD}}{0.00139 \text{ USD per chart}} \approx 719 \text{ charts per dollar.} $$
Clinical coding, not software: the codes are ICD-10-CM diagnosis codes on a patient chart.
Try it live at codesentinel-bno0.onrender.com (the first request after idle takes about a minute to wake) · Code: github.com/kishanraj41/codesentinel (MIT)
Inspiration
Risk-adjustment payment follows what the chart supports, and two kinds of mistakes eat a coder's day. A diagnosis is documented but coded too vaguely: "type 2 diabetes, uncontrolled with hyperglycemia" billed as E11.9, or diabetes and stage 3 CKD billed separately when the guidelines call for E11.22. Or a code sits on the claim with nothing behind it: a negated finding, a family history, a screening score, a problem list copied forward from 2024.
Rule engines bury the coder in review rows; a general LLM invents quotes and cites guidelines from memory. CodeSentinel gives the coder a worklist they can sign: every row backed by a real quote from the chart and, where the rules demand it, by the guideline edition in force on the day of the audit.
What it does
A coder's queue today is a pile of rule hits. Some are real missed or under-specified HCCs; the rest are family history, negated findings, screening scores and problem lists copied forward. On our 66-chart hard-negative suite, 53 of the rules queue's 132 rows are false positives of exactly that kind. CodeSentinel clears them before the coder sees the list, and attaches to each surviving row the chart quote and, for combination and specificity codes, the guideline passage the coder needs to sign it.
Pick a chart and click Audit chart. About 25 seconds later you have a coder worklist:
- Proposed gaps: missing codes, specificity upgrades (E11.9 to E11.65) and combination codes (E11.22), each with a decision, the chart quote and, where required, an official guideline citation.
- Unsupported billed codes: codes on the claim the chart does not back up.
- Dropped by Nemotron: every candidate it refused, and why: negated, family history, screening, hypothetical plan, copied forward.
- Physician query: non-leading, drafted by Ultra when documentation is ambiguous.
- Receipt: models, calls, tokens, latency and Token Factory cost for the audit.
On the featured chart, Margaret Whitmore, the note says "Her mother had CHF." CodeSentinel drops that as family history, and separately proposes her own heart failure with reduced ejection fraction (I50.22) from her own BNP, EF and medications.
The coder accepts or dismisses each row; nothing is submitted. The same pipeline is exposed as POST /audit (contract 1.0.0) and as an MCP server.
How we built it
Candidates. Deterministic rules propose gaps from labs, medications and note patterns; their queue (C0+) is the baseline.
The Nemotron cascade on Token Factory. Each stage uses the smallest model that can do the job, and configs C1 to C4 add one stage at a time so each can be measured:
| Config | Adds | Model |
|---|---|---|
| C1 | Triage of every candidate, plus the quote verifier | nvidia/Nemotron-3_5-Lightning (Nano 30B-A3B as fallback) |
| C2 | Adjudication of uncertain rows, specificity upgrades and combination codes | nvidia/nemotron-3-super-120b-a12b |
| C3 | Tavily guideline gate, verified against the official PDF | Tavily |
| C4 | Escalation of conflicts and high-value HCC upgrades, and physician queries | nvidia/Nemotron-3-Ultra-550b-a55b |
| C4a (default) | Per chart: the full C4 path if the rules propose any combination or specificity code, otherwise stop after C1 | all of the above |
Adaptive depth (C4a). The depth is chosen from the rules candidates before triage runs. A chart with any combination or specificity candidate runs the full cascade, and every such candidate goes to Super, the Tavily lookup and Ultra (on a disagreement or an upgrade worth RAF 0.3 or more), whatever triage said or how confident it was. If triage would have dropped one and the models accept it anyway, the row is held as review, never accepted. A chart with no such candidate stops after triage and the verifier: confident rows (0.8 or higher) are accepted, the rest held as review.
Every call is a strict JSON-schema request to Token Factory's OpenAI-compatible API. Provider token counts feed the receipt, so the cost shown is real usage times the catalog price.
The verifier. A quote must be a substring of the note (at least 12 characters), an exact lab or medication line, or a structured record id. Anything else is dropped as unsupported.
The demo. A FastAPI app in one container (Render free tier, via render.yaml), with inference on Token Factory. If a spend or rate cap trips, a cached run is shown under an amber banner, never passed off as live. All demo and evaluation charts are generated test records.
Third-party services: NVIDIA Nemotron on Nebius Token Factory, Tavily, Synthea, the NLM SNOMED CT to ICD-10-CM map, Render, GitHub Actions. AI tools: I used Cursor and Grok Bot as coding and writing assistants, including on this write-up. The design, the evaluation protocol and every claim are mine, and every number traces to a committed results file.
Results
All three evaluations ran live on Token Factory on 2026-10-28, on the shipped commit 8ab01be. Each compares C4a, the adaptive default, with the rules queue a coder works today (C0+).
- 66-chart hard-negative suite: precision 0.598 → 1.000 at recall 0.975, review queue −39.4% (2.00 → 1.21 rows per chart), 53 of 53 traps rejected across 11/11 categories, 3/3 unsupported billed codes caught, \$0.00139 per chart.
- 10 hand-labeled charts: precision 0.755 → 0.897 at recall 0.946, review queue 4.9 → 4.2 rows per chart.
- 300 Synthea held-out patients (I wrote no chart or label): false-positive gap rows −72% (656 → 184), gap rows per chart 2.35 → 0.75, 20 of 20 planted unsupported codes caught, and 31% cheaper per chart than the full cascade (\$0.00170 → \$0.00118).
Every stage earns its place. On the hard-negative charts the rules queue carries 53 false positives. Lightning triage and the quote verifier (C1) cut that to 2, removing 96% at the first step. Super alone (C2) leaves precision at 0.975; with the Tavily guideline passage in its prompt (C3), precision reaches 1.000. Adaptive depth matched or beat the full cascade on both hand-built sets, and on Synthea it found more true gaps than the full cascade (recall 0.683 vs 0.667) at 31% lower cost. On the hard-negative suite it costs about the same as the full cascade, so the saving shows on ordinary charts.
Full tables, confidence intervals, per-code breakdowns and known misses: eval/results/live-v9/RESULTS.md and eval/results/synthea/heldout-v9/RESULTS.md. Every run is kept, including the earlier ones.
Challenges we ran into
Testing on data I didn't write. My own charts know my blind spots. So I built a Synthea benchmark with labels derived by code from the NLM map, committed a seeded dev / held-out split before touching the cascade, and fixed only against the dev charts. Held-out runs once per code version, and every run is kept. That paid off: one fix (an "info gate" that cut review rows on charts whose notes name no condition) also hid planted unsupported codes, and held-out catches fell from 19/20 to 7/20, so I reverted it. The final code catches 20/20.
Citing the right edition. The ICD-10-CM guidelines change every October 1, and FY 2027 took effect mid-hackathon. Instead of hard-coding passages, the edition is now picked from the run date and a passage is cited only if it matches the official PDF word for word, so next October's edition needs no code change.
A platform under load. One Super adjudication call returned HTTP 529 (overloaded) after retries in an early Gold-B run, and in the first 400-chart Synthea run GET /v1/models timed out or dropped the connection on 7 charts. The client now retries 429 and 5xx with backoff, honors Retry-After, retries and caches the model list, and labels any chart where a step had to fall back as degraded so it is never counted as live. The published v9 runs have zero degraded charts.
Accomplishments that we're proud of
- 53 of 53 traps rejected (family history, negation, screening, copied-forward and seven more categories) with zero false positives on the 66-chart suite, by the adaptive default that ships.
- Every row is checkable: the chart quote, the verified guideline passage, the model that decided and what it cost.
What we learned
- Small models carry the load. Lightning plus the verifier removes 96% of the false positives on its own; on ordinary charts, adaptive depth stops there 94% of the time.
- Verification beats prompting. Checks outside the model did more for trust than any prompt change.
- Freeze the split first. It is what makes a benchmark number worth quoting.
What's next
- Certified-coder sign-off on the hand-labeled set, then grow it.
- Inpatient notes and more code families.
- A production path: Zero Data Retention, a dedicated Token Factory endpoint and the right agreements.
Best Use of Tavily
Tavily is the gate that decides whether a row can be accepted.
- Live retrieval, verified. For each combination or specificity decision, Tavily searches cms.gov and cdc.gov for the ICD-10-CM Official Guidelines in force on the run date. A passage is cited only if it appears verbatim in the official CMS PDF, with section, page and PDF SHA-256 recorded. Search results are cached, so repeat lookups for the same code family don't cost another search (the v9 runs were served from that cache: 0 new credits), and every passage is re-verified against the PDF.
- The edition follows the calendar. The target edition is chosen from the run date, so FY 2028 will be picked up the same way FY 2027 is now.
- It changes outcomes. Without a verified citation, a combination or specificity row cannot be accepted; it stays in review.
- It shows in the eval tables. C3 is C2 plus Tavily: on the hard-negative charts, with the retrieved passage in Super's prompt, precision goes from 0.975 to 1.000 (zero false positives). The gate rule itself overrode no decision in that run; what it guarantees is that no combination or specificity row is accepted uncited. Every citation in the published runs is a Tavily-retrieved passage verified in the FY 2027 PDF.
Pre-existing project: what was new in the submission period
Rebuilt from a May 2026 prototype (Ctrl+Alt+Heal, an MCP gap finder with a 33-agent runtime, now removed). New during the window: the Nemotron Lightning / Super / Ultra cascade with JSON-schema outputs and a repair retry; adaptive cascade depth (C4a, the default); the out-of-model quote verifier; the Tavily FY 2027 guideline gate with verbatim PDF verification and provenance on every citation; the POST /audit 1.0.0 contract, receipts, PHI refusal and a FHIR host allow-list; the coder web demo; and the eval harness (Gold-A, Gold-B hard negatives, and the 400-patient Synthea benchmark with the NLM SNOMED CT to ICD-10-CM map and a dev / held-out protocol). Details: docs/changes.md.
Log in or sign up for Devpost to join the conversation.