Heuristics

Every year >73M claims are denied. Most patients give up — not because they lost the appeal, but because no one helped them fight. Heuristics drafts the appeal and gets smarter with every case.

Comment

Heuristics

Inspiration

US health insurers deny roughly 19% of in-network claims — about 73 million a year on ACA exchanges alone. Fewer than 1% of those denials are ever appealed. And yet, when patients do appeal, roughly 1 in 3 wins.

That gap is the whole project.

Insurers have automated denial. They run AI-powered adjudication systems, prior authorisation algorithms, and entire denial-management operations. The patient on the other side has a thirty-page policy document, a phone tree, and — if they're lucky — a few hours of angry googling.

The inspiration was simple and uncomfortable: if the asymmetry exists because one side has better tools, build the patient a better tool. Not a chatbot that explains the process. A system that actually drafts the appeal, scores it against the denial's specific grounds, and — critically — gets measurably better at doing that every time it runs.

The project is named Heuristics publicly .

What it does

Heuristics helps any patient respond to a US health-insurance denial. The core workflow:

  1. Upload the denial letter and clinical context. Heuristics parses the denial reason, identifies the insurer, and routes to the right playbook (scoped to Aetna, Cigna, or UHC — across medical necessity and prior authorisation denial types for the hackathon submission).
  2. A question agent fills in missing context. If the denial letter is ambiguous or the clinical context is thin, a question agent asks targeted follow-up questions before drafting begins.
  3. The drafter writes the appeal. A Google ADK Student workflow — parse → insurer playbook + US-playbook → Phoenix memory retrieval → drafter → self-check — produces a full appeal letter in plain English.
  4. An outcome simulator scores the draft. A transparent rule-based proxy (rules published at eval/simulator_rules.json) scores the appeal across six dimensions: faithfulness to the denial grounds, citation accuracy, procedural compliance, clinical specificity, regulatory grounding, and tone.
  5. The Showcase Learning Observatory. This is where Arize Phoenix becomes structurally load-bearing to training. The showcase runs a full GEPA (Reflective Prompt Evolution) learning loop: score a held-out benchmark the drafter has never trained on → GEPA reads judge feedback from Phoenix traces → GEPA proposes edits to the drafter prompt, question-agent prompt, insurer slice playbooks, and US federal rules → the PM reviews the diff and approves or discards before anything ships.

Nothing promotes to production without human approval. That is not a disclaimer — it is a designed gate in the autonomy ladder.

How we built it

Stack overview:

Layer Tool Role
Agent runtime Google ADK (Python) v1 Student workflow for appeal drafting
LLM Gemini 3 Drafting, judging, reflection, question agent
Observability + evals Arize Phoenix Cloud Trace storage, eval logging, memory source
Runtime introspection @arizeai/phoenix-mcp Agent queries its own past traces at inference time
Instrumentation openinference-instrumentation-google-adk Auto-tracing for all ADK spans
Frontend Next.js (App Router) Consumer-grade UX — the showcase observatory UI
Backend FastAPI Hosts the ADK Student + showcase learning API
Hosting Google Cloud Run Two services, scales to zero

The Arize integration is not a wrapper. Phoenix serves two structurally distinct roles:

  • Training signal: Every judge run writes eval spans to Phoenix. GEPA reads those spans via Phoenix MCP to understand what failed and why before proposing edits.
  • Runtime memory: At inference time, the drafter queries Phoenix MCP for past eval patterns on similar denial types.

GEPA (Reflective Prompt Evolution) is the learning engine. It is a sample-efficient evolutionary optimizer that reads natural-language judge improvement notes and proposes targeted text edits to the prompts and playbooks that actually caused the failures. The algorithm is from ICLR 2026.

The playbook architecture is the structural differentiator. Rather than one monolithic system prompt, Heuristics maintains three learnable layers: a global drafter prompt, per-insurer × per-denial-type slice playbooks, and a US-playbook covering federal and state-scoped procedural rules. GEPA mutates the layer that actually failed — not the entire prompt stack.

The backend has 446 passing unit tests. The frontend is a full Next.js application with a live learning observatory, a promotion review modal, a diff viewer for GEPA proposals, an evidence grid (before/after learning matrix), a counterfactual memory toggle, and a case cycler with denial letters and clinical context.

Challenges we ran into

1. Phoenix as a first-class dependency, not an afterthought. Most observability integrations are read-only dashboards. Making Phoenix MCP a live inference-time memory source — where the agent actually queries its own past traces before drafting. Getting that contract right took significant iteration.

2. GEPA feedback laundering. Judge agents score drafts and leave improvement notes. But if GEPA reads those notes directly, it can overfit to the judge's phrasing rather than the underlying failure pattern. The solution is a laundering step: a reflector reads the judge notes and produces abstract improvement directives stripped of case-specific content before mutation. Getting the granularity of that abstraction right — specific enough to be actionable, abstract enough to generalise to the held-out benchmark — was a balancing act.

3. Per-slice credit assignment. When a draft scores poorly, it is not always clear whether the failure belongs to the global drafter prompt, the insurer playbook, or the US-playbook. The credit resolution module has to make that call before GEPA mutates anything. Wrong credit assignment means the wrong layer gets edited, which makes the system worse on cases it previously handled well.

4. Held-out generalisation as the gate. It is easy to get lift on training cases — that is just overfitting. The requirement of lift on held-out cases the drafter never trained on is harder. Building the held-out gate correctly — and making sure it is the thing that decides whether a GEPA proposal reaches the approval screen — required careful benchmark construction and separation of the training and holdout manifests.

5. Consumer UX on a complex technical system. The appeal path and the showcase learning path are very different surfaces. A patient filing an appeal should never see GEPA, Phoenix spans, or promotion diffs. A judge evaluating the system should see all of it, clearly. Building both surfaces — with the right information architecture for each — in a single Next.js application, with the copy and layout right for both audiences, required two lenses. There are therefore, two pages - Appeal for patients, Showcase for judges.

Accomplishments that we're proud of

Phoenix is load-bearing. The counterfactual demo is not a mock. Disabling Phoenix memory produces a measurable drop in the composite score. That is real evidence the integration is doing real work, not decorative observability.

The human-in-the-loop gate is a product decision, not a safety disclaimer. The autonomy ladder — Apprentice (human approves every promotion) → Journeyman (auto-promote on hard safety gates) → Master (relaxed gates) — is a designed product architecture. The hackathon submission ships at Apprentice. The full plan for Journeyman and Master is documented and reasoned through in the PRD.

446 backend unit tests passing. For a hackathon project, this is unusual. The drafter, judge pipeline, GEPA loop, credit resolution, playbook mutation, held-out gating, and Phoenix client all have test coverage. That discipline made the last ten days of the build much less terrifying.

The diff viewer. When GEPA proposes a change to the drafter prompt or an insurer playbook, the promotion review modal shows a full side-by-side diff with the old and new text. No black-box promotions. Every change is reviewable before it ships. This is one of the details that makes the system feel trustworthy rather than just impressive.

Built by a non-technical PM. Twenty days before submission, the builder had never read a US insurance denial letter. The whole system — ADK agent, FastAPI backend, Next.js frontend, GEPA loop, Phoenix integration — was built from scratch in that window by a non-technical PM. A testament really to how far technology has come.

What I learned

Observability is architecture, not instrumentation. The decision to make Phoenix MCP a runtime memory source changed how the entire system was designed. If Phoenix had been added at the end as a logging layer, none of the interesting parts would exist.

Playbook granularity matters more than prompt length. A 3,000-word monolithic system prompt is harder to improve than three targeted playbooks that each own their failure mode. The per-slice architecture made GEPA tractable. Without it, any mutation risks breaking things the prompt was already doing right.

The held-out benchmark is the product. Building a rigorous held-out benchmark — synthetic cases the model never trains on, scored on six dimensions, with a composite that GEPA must lift before promoting — is not an eval task bolted onto the side of the project. It is the core product decision. Everything else is infrastructure for that measurement.

Feedback laundering is underrated. The step between "judge leaves a note" and "GEPA reads that note" is not trivial. Abstracting improvement signals away from case-specific content is what makes the learning loop generalise rather than memorise. This is a pattern worth taking seriously in any system that learns from eval feedback.

What's next for Heuristics

Near-term (post-hackathon Part B):

  • Heuristics swarm. The 12-agent architecture is built and deferred in backend/app/aegis_swarm/. The swarm adds specialised agents for evidence retrieval, regulatory grounding, and medical necessity argumentation — each traceable and improvable independently.
  • 10 insurers, 7 denial types. The hackathon benchmark covers 3 insurers and 2 denial types. The full playbook library targets 10 insurers across 7 denial types, each with their own slice playbooks and GEPA-improvable rules.
  • State-scoped geo rules. The US-playbook currently holds federal rules. State-specific procedural rules (timelines, required language, state insurance commissioner contacts) are the next layer, keyed by us_state on the case.

Open questions that will drive the roadmap:

  1. Library retrieval quality. If the library agent can search online sources more freely (beyond the controlled local corpus), how much does appeal quality actually improve on held-out cases? This is unanswered and matters.
  2. GEPA cost vs lift. The GEPA loop is Gemini-call-intensive. Is the quality delta large enough to justify that cost at production cohort scale? The answer requires a production run dataset that does not yet exist.

The long-term question is whether a well-instrumented, self-improving appeal system — with Phoenix as its memory and GEPA as its learning engine — can meaningfully shift the 1-in-100 appeal rate. That is the hypothesis. The hackathon is the first experiment.

Built With

  • ai
  • arize-phoenix
  • docker
  • fastapi
  • framer-motion
  • gemini
  • google-adk
  • google-agents-cli
  • google-cloud
  • google-cloud-run
  • lucide-react
  • next.js
  • openinference
  • opentelemetry
  • phoenix-mcp
  • python
  • rank-bm25
  • react
  • rest-api
  • shadcn/ui
  • tailwind-css
  • typescript
  • uv
  • vertex
  • vertex-ai
  • vitest
  • zod
Share this project:

Updates