Track 4 Fraudulent Checks

Inspiration

Field workers inspect gas meters for corrosion, which federal pipeline-safety rules require. Inspections are tracked by volume, though, and nobody can review every form and photo. If someone logs an inspection they never did, a corroding meter could go unchecked while the safety records say everything is fine. We wanted to see whether we could find those cases with data alone. There's no answer key, so every finding had to stand on its evidence.

What we learned

One weak signal isn't enough. Several together can be. Any single flag, like a photo far from the meter or a repeated reading, usually has a harmless explanation. Fraud shows up when several flags pile up on the same worker.
Ruling out innocent explanations matters as much as detection. Meter banks, meter exchanges, poor GPS and system reload days caused most of our false alarms until we filtered them out before scoring.
Compare workers with their own company. Workers at different companies follow different norms, so we measure each worker against peers at the same company.
LLMs need guardrails to be trusted. Left alone, a model will cite evidence that doesn't exist. We had more success grounding it in a structured evidence packet and checking its output with code than with vector search.

How we built it

Detection pipeline (Python, DuckDB, pandas, scikit-learn): computes about 30 evidence signals per inspection. These include impossible travel between meters, photos taken at driving speed, backdated forms, reused photo files, barcodes that belong to faraway meters, and clockwork-regular submissions. It then scores each inspection, each worker day and each worker. Worker days also get a second opinion from an Isolation Forest model. Every flag comes with a plain-English reason.
Validation holdout: staff "Invalid survey" outcomes never feed into scoring. We keep them aside and use them only to check our results.
Agent panel (Claude): for each high-risk worker, a prosecutor and a defense (Haiku) argue from a numbered evidence packet plus a playbook of innocent explanations. A judge (Sonnet) then recommends one action: no action, spot check, audit or escalate. A code verifier rejects any decision that cites evidence IDs or numbers that aren't in the packet. A rules mode runs the same steps with no API calls.
Review console (plain HTML/JS): a ranked review queue, an animated replay map of the worker's day, a signal "fingerprint" radar against peers, prosecution and defense side by side, approve/change/dismiss buttons, routing to Legal, HR or Finance, and a full audit trail.

Challenges we faced

No ground truth. No inspection was confirmed as fraud, so we couldn't train a classifier. We relied on evidence-based rules, peer baselines and the held-out staff labels instead.
Scale. A synthetic full-year test with 1.07M inspections took about 6 minutes and about 6 GB of RAM. We added a --no-route option for machines with less memory.
Hallucinated citations. Early judge outputs cited evidence that didn't exist. That's why we built the verifier: the judge gets one retry, and then a rules judge takes over.
Privacy. The data is anonymized and must stay that way. Live mode sends only anonymized tokens and numbers to the API, and a person makes every final decision.
UI pivot. Our first interface was in Streamlit. We replaced it with a custom console so reviewers could get through a case quickly.