Inspiration

Automated decisions often arrive with a confidence score, but confidence only tells us how sure a model sounds once. FlipTrace asks a harder question: does the answer survive almost the same input?

FlipTrace is a visual stress test for AI decisions. A scenario is converted into controlled counterfactual “twins”: an identity substitution that should not matter, equivalent wording, reordered facts, small numeric movements, and boundary probes. Every twin is evaluated by the same black-box model. A deterministic audit engine then checks three things: invariance, monotonicity, and minimality. If changing only an irrelevant detail reverses the output—or improving an applicant's situation makes the model assign greater urgency—FlipTrace catches the contradiction.

What it does

The centerpiece is a flip receipt. It places the original and counterfactual side by side, highlights the only changed phrase, and shows the decision reversal in one glance. A 12-cell local decision-surface matrix, stability score, and field-sensitivity bars expose the wider pattern without hiding it inside a single metric. In the reproducible fixture, simply removing “please” changes an urgent score of 75.7 into an emergency score of 76.7—one token, no changed facts, and a reversed dispatch action.

For a safe, reproducible demo, FlipTrace audits a synthetic emergency-housing triage classifier using fictional people and generated cases. It never makes or recommends a real housing decision. The model is intentionally imperfect so the audit has realistic failures to discover, while the perturbation generator, pair matching, flip detection, monotonicity checks, scoring, and ranking are transparent TypeScript. The architecture is model-agnostic: the same audit contract can sit in front of an LLM, classifier, moderation endpoint, or rules-plus-ML pipeline.

How we built it

FlipTrace uses Next.js 16, React 19, TypeScript, Tailwind CSS 4, vinext, and Cloudflare Worker-compatible output. The public demo requires no account, database, or personal data. Audit results are computed locally from a reproducible fixture, which makes judging repeatable and keeps the safety claim honest.

Challenges we ran into

The hardest part was separating changes that should be invariant from changes that should move a decision in a predictable direction. A raw flip rate cannot distinguish a useful response from a contradiction. FlipTrace therefore attaches an expectation to every twin, measures semantic and numeric distance, and ranks the smallest violation rather than the loudest one.

Accomplishments that we're proud of

The result communicates a complex audit in seconds. The flip receipt makes a local decision boundary tangible, while the underlying engine remains deterministic, typed, and inspectable. The demo also avoids presenting synthetic output as real-world evidence.

What we learned

Fairness cannot be compressed into one score. Local counterfactual tests are most useful as falsification tools: they can reveal instability and inconsistent directionality, but they cannot prove a model is globally fair or safe.

What's next for FlipTrace

Add adapters for external model endpoints, semantic constraint checking for free-form twins, batch cohort audits, exportable evidence packets, and human-defined policy rubrics.

FlipTrace turns abstract model behavior into evidence a product team, auditor, or affected person can understand. The goal is not to certify an AI system as fair. It is to make brittle local behavior impossible to miss before people bear the consequences.

AI/ML disclosure

The submitted product audits a synthetic black-box decision model using automatically generated counterfactual inputs. OpenAI Codex helped develop and review the implementation, documentation, and submission copy. The public demo labels its synthetic fixture explicitly and does not claim that its fictional results come from a deployed housing system.

Built With

Share this project:

Updates

Submission history