Inspiration
In New York's Self-Direction program, people with developmental disabilities control their own Medicaid-funded budgets. Before that budget can pay for a riding lesson, a museum membership or a grab bar, a reviewer at a small non-profit (a fiscal intermediary) has to prove the purchase is genuinely public at a published price — by opening the provider's website, finding the price and the class, and saving date-stamped screenshots for the Medicaid audit file. If the agency approves something that didn't qualify, the money is clawed back.
I watched what that job actually looks like: one reviewer, one browser tab at a time, twenty to forty minutes per application, and an audit file that depends on how careful each person was that day. It's the kind of work an AI agent should do — but only if it can be trusted not to make things up. A hallucinated "verified" in an audit file is worse than no tool at all. That constraint became the whole design.
What it does
ProofPack takes a completed pre-approval form (PDF), reads it, visits the provider's public website, checks the things a website can actually prove, captures stamped screenshot evidence, and returns a review-ready package: report.html, report.json, a SHA-256 manifest of every capture, and the agent's full action log. A human still makes the approve/deny call. When the web can't prove something, the tool says Not Found or Needs Review — that is a correct answer, not a failure.
How I built it
Five stages, only one of them agentic:
- Extract — Gemini reads the PDF directly (digital or scanned) into a schema-validated object using a JSON response schema — no brittle text-position rules.
- Route — the form's category selects a YAML checklist. Every question is tagged
website,internalordocument; onlywebsiteitems ever reach the model. - Deterministic checks — fee caps, adults-only rules, exclusion-keyword backstop, all plain Python.
- Website research — a Gemini agent (
gemini-3.7-flash) drives Playwright through eight tools: open, read, find, list links, capture page, capture evidence, record finding, record rate comparison. I use thegoogle-genaichat API with automatic function calling disabled, so my code — not the SDK — executes every call and can refuse it. - Package — Jinja2 report, hashed evidence, run log.
The heart of the project is the three gates the model can't get past: a finding can't be "Found" without citing a capture that exists in the manifest; a quote must appear verbatim in a page the agent actually visited this session; and timestamps, URLs and hashes are burned in by Python, never written by the model. The gates live on a plain session object, so every rejection path is unit-tested without an API key or a browser, and a package auditor re-checks every committed report.
Challenges
- Trusting the agent. Early on the model would happily "record" a finding before capturing anything. Making the tool layer reject the call — and tell the model why — turned that into self-correction instead of fabrication.
- Negated requirements. "Class does not give college credits" is satisfied by a page that says nothing about credits. Getting the model to judge the requirement rather than the literal words took explicit status semantics in the prompt and a re-run to confirm.
- The web fights back. Some providers time out, some CAPTCHA headless browsers, some publish prices only behind a secure checkout. The tool detects non-content pages, never treats them as evidence, and reports Needs Review with a capture of what it saw.
- Being right about absence. I ground-truthed the negatives by hand: Gracie Barra really publishes no prices; GallopNYC really has no schedule page. An honest tool has to be right when it says "not there".
What I learned
Split deciding from evidencing. Gemini is excellent at the judgment calls — which page holds the proof, whether "contact us for pricing" is a published fee, how to explain a result to a reviewer. Python is what makes those judgments admissible: hashes, timestamps, verbatim checks. Together they produced three end-to-end reviews for $0.07–$0.19 each, one of which caught a genuine discrepancy ($85 on the form vs. $80 published).
What's next
Hosted intake (a watched Drive folder → Cloud Run queue) so agencies can use it without a terminal, explicit context caching to cut cost further, and three free pilots with NY fiscal intermediaries. Revenue today is $0; the product, the evidence and the pilot offer are in the repo.
Log in or sign up for Devpost to join the conversation.