Paper Trail

Runs Medicare’s real 1990s COBOL pricer, checks it against the law with AI agents proven on planted bugs, and prices every difference to the cent, then verifies a Python rewrite.

Comment

Inspiration

Medicare pays hospices through a COBOL pricer CMS has maintained since 1998. The payment rules live twice: once in the code, and once in the Claims Processing Manual. Every year the rules change and someone has to change this code to match, and any replacement has to prove it pays exactly the same, or nobody will switch it on. 8090's brief asks for tools that make systems like this easier to understand, improve, or replace. We wanted every step of that checked by running the real code, not by trusting a reading of it.

What it does

Paper Trail takes CMS's published hospice pricer and helps you understand it, change it and replace it, with proof at each step:

  1. Understand: it runs the real code.
    • It compiles CMS's COBOL with GnuCOBOL and prices 10,000 synthetic claims with paragraph-level traces. Every claim's receipt adds up to the COBOL's own total to the cent, 10,000 of 10,000, and version 16.0 matches CMS's own sample output 8 of 8.
    • A knowledge graph joins every COBOL paragraph, data field, business rule, manual section, run, bill and work order: over 8,000 nodes and 12,000 links. Pick a rate and it shows everything that rate touches, then reprices every bill to check.
    • Agents extract 160 business rules from the code and the manual. A citation checker verifies every one mechanically (line ranges and verbatim quotes), and none failed.
  2. Change: a rule change in plain English becomes priced COBOL.
    • One agent reads the instruction and the code and names the paragraphs that must change. A second writes the change as one-line COBOL edits. The edited program is compiled and run on the 10,000 bills and 104 edge-case bills beside CMS's code, so you see every bill and dollar that moves.
    • Four invented rule changes, so no model could have memorized them: each was planned exactly, and each implementation pays all 10,000 bills like a hand-written reference.
    • A new change typed live, the high home-care rate for 90 days instead of 60: planned, written and priced through the API in 44 seconds for $0.53 (2,637 bills, +$932,698.48).
    • Against CMS's real January 2016 reform, seeing only the rule text and the old code, the planner named the two paragraphs that had to change. It didn't name three older paragraphs the new release replaced.
  3. Replace: the rewrite, proved.
    • The FY2016 pricer and its driver, rewritten in Python, produce all 10,000 output records identical to the COBOL's, byte for byte.
    • Differential fuzzing on 100,000 generated edge-case bills: 99,400 of 99,400 compared records identical, after three fixes it forced. Seven COBOL behaviours the rewrite copies are labelled needs human review.
  4. Prove the agents before trusting them.
    • Five planted bugs, read blind: 5 of 5 caught, and each fix, compiled and repriced, restored CMS's line and its dollars exactly.
    • 961 generated one-line bugs: the 10,000 test bills catch 79.5% of those on lines they run, and 104 targeted edge-case bills raise that to 90.0%. On a random 30, blind and beside clean controls, the agents detected and repaired all 30 with no false alarms. That may flatter them: the bugs are one-line changes the manual describes, and the model may have seen CMS's public code.
    • Every rate in the pricer for FY2014–FY2021 matches CMS's published rates, 84 of 84 to the cent.
  5. Flag CMS's own code, carefully. Every candidate mismatch is judged by 3 independent critics, and only a majority stands. On the untouched code, 0 were confirmed; one disputed question is labelled needs human review, never "bug".
  6. Plan the rest of the modernization. Nine versions (FY2014–FY2021) are diffed and run to rank what changes most, and findings become 13 work orders with 305 test claims from real COBOL runs.

How we built it

  • COBOL: GnuCOBOL 3.1.2 in Docker, so every OS gets the same compiler; paragraph traces; a Python harness for the fixed-length records.
  • Knowledge graph: a COBOL parser (paragraphs, fields, PERFORM/GO TO/CALL, reads and writes) joined with manual sections, rules, runs and work orders in one validated evidence file.
  • Pricing loop: any edit is applied to a copy of the source, recompiled, re-run on the same bills and diffed bill by bill. An edit that doesn't compile gets no price.
  • Agents: Claude Opus 5.5 with JSON-schema outputs, a prompt cache, a spend cap and an offline mode. The same jobs run through the Anthropic API or through blind Claude Code subagents.
  • Web: Next.js and React Flow on a FastAPI backend.
  • Reproducibility: one command rebuilds everything offline in about 5 minutes, and every output comes back byte for byte. CI runs on every push.

Challenges we ran into

  • Getting 1990s-style IBM COBOL to compile and match CMS's sample output exactly. The FY2014 and FY2015 checks are 1 cent off on a few bills, and we report those misses.
  • A parser bug that invented 1,068 fields. We caught it when the version differ found impossible renames.
  • A real defect in CMS's driver: after an unknown provider it reads past its provider table. We proved it with subscript checking.
  • No API credits mid-hackathon. We added a second transport so the same jobs could run through Claude Code without changing a single prompt.

Accomplishments that we're proud of

  • Real code, not a toy: CMS's actual published COBOL, running and matching CMS's own sample output.
  • Checked by running the code: every dollar on screen comes from a real run, never from the AI.
  • The AI is tested before it's trusted: planted bugs, hundreds of generated ones and invented rule changes, with the limits written next to the numbers.

What we learned

The most useful thing an AI agent can do on legacy code is make claims you can check, not read the code more cleverly. Every rule cites lines that are verified, and every fix is priced by running the code. Every finding on code we didn't plant is labelled for a human, not declared a bug.

What's next

  • Bigger changes. Today the writer makes one-line edits, so a rule that needs a new field or paragraph is only planned, not written.
  • Two independent builders. One edits the COBOL and one edits the Python, and they must agree on every bill.
  • A rewrite agent that translates one paragraph at a time and proves each with the bills and the fuzzer.
  • More CMS pricers (ESRD, SNF), which share the same archive structure.

Built With

Share this project:

Updates

Submission history