Appeal Pilot

Appeal-Pilot turns insurance denials into chart-cited, fact-checked appeal letters using NVIDIA Nemotron agents on Nebius, so clinicians just review and sign.

Comment

Inspiration

Picture a patient whose knee MRI gets denied because "conservative therapy was not documented." The chart actually shows six weeks of physical therapy, but it's on page 14 of the notes, and no one at the clinic has an hour to dig it out and write an appeal. So the patient waits, or gives up.

That isn't rare. In 2024, Medicare Advantage insurers denied 4.1 million prior authorization requests, and only 11.5% were appealed, even though 80.7% of appeals won. Physicians already spend about 13 hours a week on prior authorization. We built Appeal Pilot to find that evidence on page 14, cite it, and hand the clinician a letter they can trust.

A good appeal is tedious work that has to be done carefully. You read the denial, find the payer's coverage policy, match each criterion to evidence in the chart, and write a letter that never overstates that evidence. That seemed like a good fit for AI agents, as long as we could make the output trustworthy. A letter that misstates a patient's chart can do more harm than sending no letter.

What it does

Appeal Pilot turns a prior authorization denial into an appeal letter that's ready for review. A clinician signs off on every word.

  • Upload a denial letter and chart notes (PDF or text), or pick one of 10 sample cases.
  • Watch the agents work, with live progress on screen. They read the denial, retrieve the payer's coverage criteria, check each criterion against the chart, and draft the letter.
  • Trace every claim. Each criterion is marked met, not met, or unclear, with the chart lines as evidence. Every sentence in the letter cites its sources.
  • Review the fact-check. An independent reviewer agent flags any sentence the chart doesn't support. The clinician keeps or removes each flagged sentence and can edit any sentence.
  • Approve and download the final letter.

How we built it

Models: All five agents run on NVIDIA Nemotron through Nebius Token Factory, our only model provider. Nemotron-3-Nano-30B-A3B handles fast extraction (reading the denial and pulling out policy criteria). Nemotron-3-Super-120B-A12B handles the reasoning-heavy steps: matching, writing, and review.

Agent pipeline: Intake → Policy → Matcher → Writer → Reviewer → rule checks. All agents return structured JSON validated with Pydantic schemas.

  • Policy retrieval: We built a corpus of 15 coverage policies adapted from real public policies (CMS, Aetna, BCBS, and others). Each file cites its source. Policies are split by section and indexed with embeddings, falling back to BM25 if embeddings aren't available. Retrieval is filtered by payer and prefers the policy ID cited in the denial. The policy agent keeps criteria from only a single retrieved policy, so the letter can't argue from an invented or mixed source.
  • Citations: Chart notes are split into numbered, citable lines (L1, L2, …). The writer must attach line IDs, criterion IDs, or DENIAL to every factual sentence.
  • Rule-based fact-checking: A no-LLM check pass catches unknown citation IDs, factual claims without a citation, and any number or date that doesn't appear in the cited sources.
  • Stack: Python, FastAPI, Pydantic, the OpenAI-compatible SDK pointed at Token Factory, rank-bm25 and NumPy, and a single-page HTML frontend. Docker is supported.

Challenges we ran into

Getting reliable structured output from reasoning models. Responses came back wrapped in blocks or code fences, and not every endpoint supported JSON-schema mode. We wrote a cleaning layer, fell back to JSON-object mode when needed, retried with backoff on rate limits and 5xx errors, and added a validation retry that feeds the errors back to the model.

False positives in the fact-checker. The first version of the rule checks flagged things like "criterion 2.2" (a policy section number) or inline "(L4)" markers as unsupported numbers, and it missed dates. Tuning the checks to catch real fabrications without overwhelming the clinician took several iterations. Matching the right policy. The intake model wrote payer names in many forms, such as "… Plan" or "Northwind Care, Inc." We needed matching that tolerated those variations without ever pulling another payer's criteria.

Being honest when the evidence isn't there. The easy path was to claim support for every criterion. Making the system switch to a peer-to-peer request when evidence is missing took explicit design in both the prompts and the checks.

Accomplishments that we're proud of

  • Every claim is traceable, from letter sentence to chart line to policy criterion.
  • Two independent safety layers. A separate reviewer agent and the rule-based checks back each other up. Neither one is trusted alone.
  • The clinician stays in charge. The app drafts, flags, and explains, but only a person can approve, and nothing leaves the system automatically.
  • Grounded in real data. The policies are adapted from real public policies, and the denials follow real IMR decision patterns, not made-up examples.

What we learned

  • With medical and legal documents, checking the output matters more than generating it. Most of the engineering went into making sure the letter is correct, not into making it sound good.
  • Matching model size to the task pays off. The smaller Nemotron Nano model handles extraction quickly and cheaply. The larger Super model was worth using for nuanced matching and review.
  • Simple rule-based checks catch what LLM reviewers miss. Checking a number or date against its source catches a wrong date immediately. An LLM reviewer might not.

What's next for Appeal Pilot

  • Live payer policies, with a larger, automatically updated policy corpus.
  • Payer-specific appeal formats and deadline tracking, including alerts before the appeal window closes. Learning from outcomes: track which appeals are overturned and feed that back into how arguments are framed.
  • Clinical pilots with practices and billing teams to measure real-world impact, along with the compliance work (HIPAA, BAAs) needed for real patient data.
  • Train on real patient cases to learn which citations and pieces of evidence persuade payers most, and use that to improve how the letters are built.

Built With

Share this project:

Updates

Submission history