Lookback

When one person wins an appeal because the software was wrong, Lookback fixes the bug and finds everyone else it harmed. It acts alone only in a person's favor.

Comment

Inspiration

A woman we'll call Ana gets food assistance (SNAP). One month her benefit drops by $19. The software that calculates it allowed her $35 for medical costs instead of the $150 the rules say she is owed. She takes a day off work, waits six weeks for a hearing, and wins. Her money comes back.

Her neighbor Ben has the same medical costs and the same cut. He never appeals. Nobody tells him the software was wrong, so he stays short every month.

Almost everyone is Ben. In Colorado, about 111,500 SNAP denials, cuts and overpayment claims in one year led to just 262 hearings, about 0.2% (Colorado Center on Law and Policy). In Medicare Advantage in 2024, 4.1 million prior authorization requests were denied, only 11.5% were appealed, and 80.7% of those appeals won (KFF). When people do fight, they usually win. Most never fight.

The law already says what should happen next. Federal rule 7 CFR 273.17 says a state agency "shall automatically take action to restore any benefits that were lost," with "no action by the household" needed. But only for errors the agency discovers, and only twelve months back. Every month a bug goes unnoticed, families lose that month for good. Research on public-benefits software reaches the same conclusion: these systems fail quietly, and appeals and lawsuits are how failures come to light (Public Administration Review, 2026).

So we asked: what if one person's win automatically fixed it for everyone?

What it does

Lookback turns a won appeal into a bug report, and the bug fix into a remedy for everyone the bug harmed.

  1. An appeal is won. A hearing officer records the decision on a GitLab work item and mentions Lookback.
  2. Is it the software? A GitLab Duo agent reads the decision and the code and decides whether this was a software error. A second agent writes a test from the person's real facts and checks that the current code gets it wrong. No reproduction, no change.
  3. Fix it safely. A third agent fixes the code, cites the regulation, and opens a merge request. Every check must pass: security scans, a purity check on the rules code, and a replay of the last 90 days of real decisions through the new code. Then a person clicks merge.
  4. Look back. After the merge, Lookback searches the code's history for the release that introduced the bug and re-runs every decision that release made, now with the corrected rules.
  5. Make it right. It works out what each harmed family is owed under the twelve-month limit in the law, records each correction in a tamper-evident ledger, and publishes the batch as a GitLab release for the agency to pay.

It can only act in a person's favor. If the fix shows someone was overpaid, Lookback takes no action. It opens a review item for a caseworker and stops. No agent has a tool that could create a debt or close a case. We call this asymmetric autonomy. It matters because automated debt systems have caused national scandals: Australia's Robodebt raised 470,000 unlawful debts that the government had to refund (The Guardian, 2020).

Bring your own decisions. Anyone running a benefits system can upload past decisions, with the household facts and what was paid, on the Try page or through the API. Lookback returns who was underpaid and by how much, who was wrongly denied, which months are past the twelve-month limit, and which overpayments need a person. Nothing is stored, and no account is needed.

Every result is explainable. Ask "why does this household get $42?" and GitLab's Duo CLI calls Lookback's rules engine through MCP and answers with thirteen steps, each citing its regulation. The agent is not allowed to do the math itself.

It already ran, twice, on this project

Run 1, an underpaid medical deduction. From the hearing officer's comment to an opened merge request took 3.5 minutes. After the merge, Lookback replayed 10,656 past decisions and found 413 households underpaid, $25,861 owed. Nine households turned out to be overpaid. Lookback did nothing to them and opened nine review items for a caseworker. Adverse actions taken: 0.

Run 2, a family wrongly denied. A different bug, never covered by any test: the code looked up the income limit for a household one size too small. The agent found the line unaided. Lookback found all 19 families wrongly denied, $4,616, including months after their cases had been closed.

Two runs where the right answer was to do nothing. The first attempt on run 2, before the hearing text was in place, returned "unclear" and changed nothing. Later we re-ran the agents on that already-fixed appeal: they ran the test, saw it pass, and opened nothing.

The households are real: USDA's public audit sample of SNAP cases (FY2024, de-identified). The hearing decisions are written by us, modeled on published New York decisions, and labeled as constructed everywhere.

How we built it

On GitLab, across all nine lifecycle stages:

Stage What Lookback does there
Plan The appeal becomes a work item; a mention starts the flow in one second
Create Three GitLab Duo agents in a custom flow from the AI Catalog: triage, reproduce, fix
Verify Every merge request replays 90 days of real decisions; an unexplained change blocks it
Package One container image per release, signed (Sigstore), kept so the past can be replayed
Secure Code, secret, dependency and container scanning; agents run in a sandbox that can reach exactly one server
Release Deploys to Google Cloud Run; each remediation batch is its own GitLab release
Configure The law's remedy rules and SNAP's yearly amounts live in the repository with their sources
Monitor A nightly job re-checks every recorded decision against the live engine and flags drift
Govern Tamper-evident ledgers, a chat agent and an MCP server that answer "why", all read-only

The rules engine is plain code, not AI. It calculates in whole cents, and every step cites its regulation. If a month has no official amounts loaded, it refuses to guess.

The ledger records every decision with the exact code version that made it. A fix never rewrites history; it adds a correction that points back. Anyone can verify the chain.

Google Cloud: Cloud Run hosts the rules engine, the MCP server and the website. Artifact Registry holds the signed images, and Cloud Storage holds the ledgers. The pipeline deploys with no stored keys (Workload Identity Federation).

Live: the website, the rules engine and MCP server, and the repository. Every number on the site links to the command that produces it.

Challenges we ran into

  • The twelve-month limit bit us first. Our first dry run restored $0, because the audit data was older than twelve months. That was the law working as written. We shifted the demo months forward and recorded the shift in every ledger record.
  • Approval steps inside a flow crashed. GitLab's in-flow approval step failed on every resume. We moved the human decision onto a work item instead, which turned out to be a better audit record.
  • Agents could not run tests at first. The default agent image had no way to install our dependencies. We built our own agent image on top of GitLab's, so the agents now run the real tests.
  • The tool server worked only from the command line. Lookback's MCP server works from GitLab's Duo CLI, but not yet from Duo Chat on the web or from automated flows.
  • The merge stays a human click. GitLab's agents have no merge tool, so we declared Supervised rather than claim a loop with nobody in it.

We reported what we found to GitLab: a reproduction on gitlab-org/gitlab#632998 and two new issues, #634092 and #634093. Another participant confirmed our diagnosis on the first within minutes.

Accomplishments that we're proud of

  • It works on real data. Our engine matches USDA's own calculated benefit on 99.0% of 42,327 real audit households.
  • It caught a bug nobody planted. An open issue in a public rules engine (rulespec-us #1394) silently reuses last year's amounts. Run on the same real households for October 2026, 92% would be underpaid. Scaled with USDA's weights, that is about 19 million households and $324 million in one month.
  • Zero harm, measured. Across 14,030 harmed decisions found, Lookback took 0 adverse actions and held all 5,180 overpayments for a person. We also tried to trick it with 30 hostile hearing decisions, asking it to create debts, close cases or merge its own code. None of those actions are possible: no agent holds a tool for them.
  • Light on energy. Finding the 413 households took about the energy of one CI job (0.05 Wh), because replay is plain code. Asking an AI model about each past decision would have taken an estimated 6,400 Wh, roughly 121,000 times more.

What we learned

  • The hard part is the remedy, not the fix. Fixing a bug stops future harm. Finding and repaying the past is where the law, the data and the engineering meet.
  • Let AI write code, never compute money. Agents are good at reading a hearing decision and finding the wrong line. Amounts belong to tested, deterministic code that cites the law.
  • Safety by construction beats safety by prompt. The strongest guarantee we have is that the tool to harm someone does not exist.
  • Say what is constructed. Labeling our made-up hearing decisions, and every limit, made the real results more credible, not less.

What's next for Lookback

  • A pilot with a state agency or a legal aid office, starting with one rule area and real hearing outcomes.
  • More programs. Anything that records decisions with their inputs and a code version can be replayed: Medicaid, unemployment insurance, and outside government, payroll, billing and insurance claims.
  • Payment hand-off. Today Lookback records what is owed; next it would hand the batch to the agency's payment system, with a person signing off.
  • Fully hands-off, once GitLab adds label triggers, MCP tools in automated flows, and a merge-when-green tool for agents. Each of these is one of the issues we filed.

Path A, Supervised. Not legal advice.

Built With

  • astro
  • docker
  • fastapi
  • gitlab
  • gitlab-ci
  • gitlab-duo-agent-platform
  • google-artifact-registry
  • google-cloud
  • google-cloud-run
  • model-context-protocol
  • nginx
  • python
  • sigstore
  • workload-identity-federation
Share this project:

Updates

Submission history