Ariadne

Ariadne is a decision governance layer for high-risk ML systems: models provide evidence, deterministic rules produce verdicts, and every decision is sealed and traceable.

Comment

Ariadne — Devpost submission

Submitted to **The Agent Hackathon (DataHub)* — Challenge 3: Production ML Agents.* This file is written to be pasted directly into the Devpost submission form (datahub.devpost.com). Section headers match Devpost's story fields.

Elevator pitch

Ariadne is a decision-governance layer for high-risk ML systems: a model provides a signal, deterministic rules corroborate it, and every verdict is sealed into a tamper-evident ledger and written back to DataHub — so no BLOCK ever rests on "the neural network said so," and no model can be silently swapped without the catalog catching it.

Inspiration

Anti-money-laundering systems are exactly the high-stakes ML case where "trust the model" is not an acceptable answer. A missed launderer costs money; a false positive freezes a legitimate customer's account; and under regulation like the EU AI Act, the deployer — not the model vendor — carries the liability for a decision it cannot explain. Most AML pipelines ship a raw risk score and stop there. A score is a number, not a reason, and a metric-monitoring dashboard will tell you a distribution moved without ever telling you whether one specific decision was defensible.

The scenario that crystallized the project: someone retrains a model, redeploys it, and tells no one. Predictions change overnight. Nothing alarms. By the time anyone notices, money is already gone. That failure mode — the silent model redeploy — is exactly what Challenge 3 (Production ML Agents) asks agents to catch, and it is exactly what a metrics dashboard is structurally blind to, because the dashboard trusts the model's identity instead of verifying it. We wanted a system where that trust is never assumed — it's checked, against a catalog, every time.

What it does

Ariadne sits between a GraphSAGE model scoring the IBM Anti-Money-Laundering transaction graph and the action that gets taken on its output:

  1. Keeps the model out of the decision. The GNN's continuous risk score is discretized into a band and never re-enters the decision path as a float. A corroboration gate combines that band with explicit, dataset-grounded rules — fan-out beyond a threshold, laundering-cycle membership, cross-currency layering. A transaction can escalate to BLOCK only when the model and at least one explainable rule agree. The model alone can never pull the trigger.
  2. Produces one of four auditable verdicts — PASS / WARN / REVIEW / BLOCK — with rule weights as exact Fractions, not floats, so the reasoning is exactly reproducible, not just approximately so.
  3. Seals every verdict into an append-only, tamper-evident SHA-256 hash chain (audit_hash = sha256(canonical(payload) + prev_hash)) that anyone can independently recompute with nothing but the Python standard library — no trust in Ariadne's own code required to verify its output.
  4. Reads DataHub for context, and writes back to it. A single bridge module reads a dataset's schema, owners, and upstream lineage from a live DataHub catalog to build rule context from real metadata instead of hard-coded column names, then writes each sealed verdict back as an INFERRED assertion whose externalUrl points at the audit hash. The catalog is not just consulted — it's contributed to.
  5. Catches the silent break. A model-health check compares a fingerprinted training baseline — read back from DataHub's own ML lineage (mlModelProperties) — against what's actually deployed, and seals one of HEALTHY / DEGRADED / BROKEN / ABSTAIN. A changed fingerprint or a dropped feature comes back BROKEN. When there isn't enough to compare, it says ABSTAIN — an honest "we don't know," never a false-confident "healthy."

The AML use case is a demonstration, not the ceiling: the same corroboration engine — model signal + explainable rule + sealed verdict — applies unchanged to a SIEM risk score gated before auto-containment fires, a forensic classifier's finding sealed against its evidence chain, a credit or insurance decision, or an SRE anomaly paging an on-call. Swap the rules, keep the ledger.

How we built it

  • Training / graph: build_graph.py builds a transaction graph from the IBM-AML CSVs; train_gnn.py trains a GraphSAGE encoder + edge classifier to predict "Is Laundering" per transaction, handling the ~0.1% positive-class imbalance with pos_weight and selecting the checkpoint by validation AUPRC.
  • The decision core (ariadne/), all pure Python, unit-tested, framework-agnostic: score_discretization.py maps the float score to a discrete band (the float dies there); rules.py implements the explicit, Fraction-weighted rules; verdict_engine.py is the corroboration gate itself (ADR-001); canonicalize.py gives stable byte serialization for hashing; audit_chain.py is the SQLite-backed tamper-evident hash chain; model_health.py is the silent-break detector.
  • ariadne/datahub_bridge.py is the only module that talks to DataHub — built on the DataHub Python SDK (DataHubGraph for reads, DatahubRestEmitter for writes). It reads dataset schema/owners/lineage and the model's registered fingerprint, and writes dataset↔model lineage plus INFERRED verdict and model-health assertions.
  • Agent interface: ariadne/mcp_server.py (FastMCP over stdio) exposes score_transaction, explain_verdict, check_model_health, check_model_health_from_catalog, verify_ledger, resolve_review, and list_open_reviews as MCP tools, so any MCP-capable agent — Claude, Claude Code, a Slack bot — can call the same engine a human would.
  • Two UIs on the same API: a Next.js app (live at web-nu-rosy-44.vercel.app, backed by a real deployed API and a real public DataHub instance) and a one-screen Streamlit app for a pure-Python, dependency-light local demo — move the model's band live and watch REVIEW flip to BLOCK the instant a rule corroborates it.
  • Closing the loop on real data: run_batch.py loads the trained checkpoint, scores a real slice of HI-Small transactions, discretizes, runs the gate, seals every verdict, and (optionally) writes the assertions back to a live DataHub — the entire pipeline, not just the deterministic layer in isolation.
  • Verified against a live DataHub quickstart, not just offline fakes: schema, owners, and lineage round-tripped for real; verdict, model-health, and training-data-lineage assertions all emitted against a running GMS instance (:8015).

Challenges we ran into

  • A float that refuses to stay out of the decision. The natural instinct with a GNN is to threshold its raw score directly. We deliberately discretized it into a band before the gate ever sees it, and pushed every rule weight to exact Fractions — so "why did this block?" always has a bit-exact, reproducible answer, never a floating-point one a second run might not reproduce.
  • DataHub rejected our first lineage write — and that was the right outcome. Registering model→training-dataset lineage naturally suggests the dataset-only upstreamLineage aspect; GMS correctly returned 422 Unknown aspect when we tried it on an mlModel. Live validation against a real server caught a bug our offline test fakes could not, since the fakes happily accepted whatever we sent. We fixed it to the correct mlModelTrainingData aspect, added a regression test, and wrote up the friction as a proposed docs contribution back to datahub-project/datahub (docs/OSS_CONTRIBUTION.md) — a clearer "which aspect for which entity" table and a more informative 422 error body.
  • A tamper-evident chain that only checked five entries back. A red-team pass on our own audit chain found that append() only re-verified the last five rows before sealing a new one — meaning an attacker with file access could corrupt row n-6 and still get later entries sealed on top of a broken chain. We fixed append() to re-verify the entire chain (linkage and hash integrity) before every seal, at the documented cost of O(n²) over a ledger's lifetime — an accepted trade-off at the intended scale of hundreds to low thousands of decisions per day. The same audit pass found and fixed 31 further findings (indexing DoS risks, a silently overwritten timestamp field, an overly broad exception handler on the MCP path) — all regression-tested; see docs/RED_TEAM_FIXES.md.
  • Making "we don't know" a first-class answer, not a bug. When a rule's required input was never computed — a 72-hour fan-out window nobody ran, for instance — the tempting shortcut is to treat the missing value as zero and move on. We made the rule abstain instead, sealing the abstention alongside the verdict so it's visible, not hidden. The same discipline drives model_health's ABSTAIN verdict: when the facets needed for a comparison aren't available, the honest answer is "cannot tell," never a false HEALTHY.

Accomplishments that we're proud of

  • A BLOCK justified solely by a neural network is structurally impossible. Not a policy, a code path: the corroboration gate requires a firing rule and a strong model band before the highest-severity action fires — verified on real batch runs, not just unit tests.
  • The corroboration gate holds at scale, on real data, not just in a curated demo. A 200-transaction batch run against the trained GNN on real HI-Small transactions produces {PASS: 42, WARN: 8, REVIEW: 148, BLOCK: 2}: the model wants to escalate 148 transactions, but only the 2 with a real corroborating rule (a genuine 2-hop laundering cycle) become BLOCK. That shape is the thesis, on data the model has never seen curated for a demo.
  • Every decision is independently recomputable, with no dependency on trusting Ariadne's own code: verify_ledger() — or sha256(canonical(payload) + prev_hash) by hand with the standard library alone.
  • DataHub is read from and written to, for real, against a live quickstart, not mocked: schema/owners/lineage round-trip in, verdict/model-health/lineage assertions round-trip out, with the externalUrl on each assertion pointing back at the sealed audit hash — so a reviewer can go catalog → assertion → hash → recomputed verdict, trusting nothing along the way but arithmetic.
  • The silent-redeploy problem — the literal Challenge-3 prompt — is caught before it costs money, sourced from the catalog's own registered fingerprint rather than a value hard-coded into Ariadne.
  • 95 tests, a live-server validation pass, and a 32-finding red-team audit, every finding fixed with its own regression test — for a project whose entire value proposition is "you can trust what this seals."

What we learned

  • Governance is a design constraint, not a feature to add later. Deciding upfront that "the model alone can never BLOCK" shaped the architecture from the first commit — the discretization boundary, the Fraction weights, and the corroboration gate all fall out of that one sentence.
  • Offline tests and a live server catch different classes of bugs. Our fakes happily accepted an invalid DataHub aspect; only a real GMS instance rejected it with the correct error. Contract bugs need a contract to fail against.
  • "We don't know" is a harder verdict to design for than "yes" or "no," and a more honest one. Building ABSTAIN into both the rule layer and the model-health check as a real, sealed, first-class outcome — not a caught exception — took more thought than adding another true/false branch, and it's the piece we'd defend hardest to a regulator.
  • A tamper-evident ledger's failure modes are subtle and worth red-teaming your own code for. The five-entries-back tail check looked like real verification until we asked "what if the attacker targets entry six."

What's next for Ariadne

  • A time-windowed fan-out computation, so the 72-hour fan-out rule can fire honestly instead of abstaining on an unwindowed value.
  • Numeric feature-distribution drift in the model-health check, banded to stay out of the sealed decision path the way every other signal already is.
  • Packaging the engine as a DataHub Skill manifest so it shows up directly in the catalog's own agent tooling, not just as an external MCP server.
  • Porting the ledger from SQLite to PostgreSQL / a distributed log, keeping the same stdlib-recomputable hash-chain guarantee at higher throughput.
  • Opening the documentation PR to datahub-project/datahub proposing a "lineage by entity type" table and a more informative unknown-aspect error — drawn directly from the bug we hit and fixed (docs/OSS_CONTRIBUTION.md).

Built with

python · pytorch · torch-geometric (GraphSAGE) · acryl-datahub (DataHub SDK) · fastmcp (Model Context Protocol) · sqlite · sha-256 · streamlit · next.js · vercel · datahub

Try it out

  • Live app (no install): https://web-nu-rosy-44.vercel.app — move the model's band and watch REVIEW flip live to BLOCK the instant a rule corroborates it; try "Simulate silent redeploy" to see a real BROKEN model-health verdict sealed and written to DataHub in real time.
  • Live DataHub instance (datahub / datahub): http://54.227.49.15:9002 — dataset ibm_aml.HI-Small_Trans → Quality → Assertions shows every verdict sealed from the live app, as it happens.
  • API health check: https://54-227-49-15.sslip.io/api/health
  • 30-second offline demo, no GPU, no dataset: bash pip install acryl-datahub PYTHONPATH=. python examples/generate_examples.py Writes four sealed verdicts, a verified tamper-evident audit ledger, and the DataHub assertions to examples/.
  • Repository: https://github.com/annatchijova/ariadne (Apache-2.0)

Notes for judges

  • New-project disclosure: every line of code in this repository was written from scratch during the hackathon's Submission Period — the repository's first commit is 2026-07-23, full history is public on GitHub. No pre-existing code or modules from another project are incorporated; standard third-party libraries (PyTorch, torch-geometric, acryl-datahub, FastMCP) are used as ordinary dependencies.
  • Tracks touched beyond Challenge 3: the MCP server makes Ariadne callable infrastructure for any agent (Track 1); read_dataset_context already drives rules from live catalog schema instead of hard-coded columns (Track 2); the tamper-evident hash chain is a general, stdlib-verifiable audit primitive usable outside AML entirely (Track 4).

https://github.com/annatchijova/

Built With

Share this project:

Updates