Ariadne: Verifiable AI Explanations

AI explanations into testable claims, runs controlled interventions against real model versions, and automatically re-audits them reporting supported, contradicted, or inconclusive evidence.

Comment

Inspiration

AI systems often explain their decisions with confidence, but a plausible explanation is not necessarily a faithful one.

We wanted to make explanations testable. If a model says “urgency marker was the primary driver,” Ariadne asks: does changing that feature actually change the model’s output more than a control feature does?

The name comes from Ariadne’s thread: a traceable path through a difficult maze of model behavior, evidence, and uncertainty.

What it does

Ariadne converts natural-language AI explanations into structured claims and tests them through controlled interventions.

It:

  • Extracts a testable claim from an explanation
  • Designs baseline, intervention, and control experiments
  • Measures effect size, validity, reproducibility, and model stability
  • Produces only three verdicts: SUPPORTED, CONTRADICTED, or INCONCLUSIVE
  • Scopes each verdict to a specific model version and data distribution
  • Maintains append-only, hash-chained evidence lineage
  • Automatically schedules re-audits after model deployments or distribution changes
  • Escalates high-impact decisions for human review

How we built it

Ariadne uses four bounded roles:

  • Investigator: interprets the explanation and extracts a claim
  • Experimenter: designs the intervention and records evidence
  • Verifier: deterministically computes the verdict
  • Governor: applies policy and decides whether review is required

Gemini is used for semantic reasoning, but never for the final verdict. The Verifier has no language-model access and computes its result entirely from recorded measurements.

The system is local-first, with offline adapters for the event bus, database, runtime state, and reasoner. The cloud deployment uses React, TypeScript, FastAPI, Google Cloud Run, Pub/Sub, Firestore, Cloud SQL, PostgreSQL, Vertex AI, Docker, and Terraform.

Ariadne architecture

Ariadne system flow

Challenges we ran into

The hardest challenge was preventing an AI agent from judging its own work. We created explicit authority boundaries and tests that verify the Verifier cannot load model-calling code.

We also had to handle duplicate events, worker crashes during experiments, malformed model responses, unstable model outputs, distribution shift, and the possibility of auditing the wrong model. Ariadne now fails closed when the target model cannot be identified or validated.

Another challenge was designing a system that could honestly return “I don’t know.” An invalid or weak experiment should produce INCONCLUSIVE, not a manufactured contradiction.

Accomplishments that we're proud of

We built a complete event-driven audit pipeline that runs locally and is deployed on Google Cloud with real Pub/Sub, Firestore, Cloud SQL, and Cloud Run.

Ariadne audited live Vertex AI Gemini model versions and found that the same explanation could receive different results from different versions.

In our transparent synthetic laboratory, where the model formulas are published and every result can be checked by hand, the full protocol achieved 14/14 accuracy.

We also ran 24 adversarial explanations. None produced a false SUPPORTED verdict, although the benchmark is intentionally small and is not evidence that Ariadne cannot be fooled.

Most importantly, the system treats INCONCLUSIVE as a legitimate scientific result rather than a failure state.

What we learned

We learned that an explanation is not simply true or false. It is true, contradicted, or unresolved for a particular model version, distribution, and intervention protocol.

We learned that control arms and validity checks matter. Removing the validity gate caused false contradictions in our benchmark.

We also learned that model instability can be larger than the effect being measured. Repeating identical requests and measuring the model’s noise floor is essential before trusting an observed effect.

Finally, we learned that honest limitations are part of the product. Ariadne measures behavioral explanation faithfulness; it does not prove universal causal truth or clinical validity.

What's next for Ariadne: Verifiable AI Explanations

Next, we want to:

  • Audit more real-world model providers and model types
  • Expand the benchmark with a larger held-out evaluation set
  • Improve claim compilation so ambiguous explanations are surfaced earlier
  • Add integrations with production model registries and drift monitors
  • Build richer human-review workflows for high-impact decisions
  • Replace the current always-on worker deployment with a production-grade Pub/Sub push architecture
  • Support domain-specific intervention protocols with versioned safety policies

The long-term goal is simple: make AI explanations measurable, versioned, and accountable instead of merely persuasive.

Built With

Share this project:

Updates

Submission history