Predictive-Powered Inference

An engine with an active statistical learning loop, wrapped in a dashboard that runs live experiments over real state DOT work-zone.

Comment

Inspiration

Government agencies are drowning in AI predictions they can't legally or scientifically trust. A DOT can ask an LLM to read every work-zone report in the state — but averaging a million biased guesses produces a confident wrong answer. Prediction-powered inference (PPI), from recent Berkeley statistics research, offers a way out: use AI predictions for speed, use a small verified sample to measure and cancel the AI's bias, and report error bars that are provably honest. We wanted to turn that math into a working government prototype — live data, real feeds, hostile auditing — not a notebook demo.

What it does

The engine answers questions like "how long does a typical work zone last?" from live state DOT feeds. An ingestion agent normalizes WZDx feeds from Mississippi, Utah, Missouri, and Kentucky (1,800+ real records, three schema dialects) with per-field provenance. A labeling agent (Claude, Gemini, or a deterministic heuristic — identity always displayed) predicts answers from free text only. A verification agent grounds truth from authoritative fields and refuses what it can't ground. An active-learning policy spends a fixed label budget where it shrinks error bars fastest, and the dashboard streams the corrected estimate live — with a 3D scene showing the point cloud, the confidence-interval slab tightening as budget is spent, and a heatmap of where the policy is looking. Every number traces to the Python statistical core; classical and random baselines are always shown.

How we built it

Python statistical core (numpy/scipy) implementing rectified and power-tuned PPI estimators for means, quantiles, OLS, and logistic regression, validated against a separately-reviewed golden reference at 1e-10. Next.js + TypeScript + React Three Fiber frontend; Vercel serverless with chunked, resumable, byte-identical runs. Everything passed a "gauntlet loop": build, automated gate, then a separate hostile critic agent that re-derives the math and drives the live app, iterating until written sign-off — nine signed audit logs.

Challenges we ran into

The hardest: active selection silently breaks PPI's guarantee. Our critic constructed an attack driving a 95% interval to 13% coverage. The fix — anchoring the estimator to the full pool's prediction mean so selection can't tilt it — restored 96%, and that attack is now a permanent regression gate. Also: three WZDx dialects that disagree with their own spec, duplicate feature IDs, states publishing "unknown" for every impact field, and serverless time limits forcing a fully resumable state-machine design.

Accomplishments that we're proud of

A coverage simulation as a build gate: 7,960 seeded replications; any 95% interval covering below 93% fails CI. Bit-identical results between local Windows and Vercel Linux through every round and bootstrap. Intervals half the width of classical sampling at the same budget — and a gated demo where the naive AI-average approach covers the truth 0% of the time while PPI covers 95.6%.

What we learned

Never trust a model — measure it. The same discipline applied to our own code: hostile review found real statistical defects polite review would have missed. Honest documentation of gaps beats polished concealment; every "known limitation" section survived audit.

What's next for Predictive-Powered Inference

Human-in-the-loop labeling with real cost tracking, more states and estimands (crash severity, pavement condition), server-side multi-user run storage, and exact finite-sample methods for adaptive designs.

Built With

Share this project:

Updates