Ariadne

Inspiration

Whilst building ML models for this hackathon, we kept running into the accuracy changing (sometimes higher other times slightly higher) but we found no tools that told us why. ML monitoring says a model moved but never why, because it cannot see the warehouse. DataHub holds that chain and does not watch the model. The gap between them is a graph.

What it does

Ariadne walks column lineage from raw table to deployed model. Names restricted attributes that got in, measures what the model can still rebuild about people, and files findings as DataHub incidents.

How we built it

We used census data to Postgres, dbt marts, MLflow registry, all catalogued by DataHub's own postgres, dbt and mlflow connectors. Agents read it back through MCP and the Agent Context Kit.

Challenges we ran into

Tags land on the dbt sibling, not postgres, so checks silently pass. The graph index returns an empty graph for 40s after ingest. Postgres must be ingested twice, before dbt and after mlflow.

Accomplishments that we're proud of

The pivot was very sudden but breathtaking once we fully built it out. Seven hop column lineage, every edge from a real connector. 15 reconstruction measurements over 244k rows. One column open under two statutes at once. A demo that re-runs live on the instance. All in record time

What we learned

A check that finds nothing looks identical to one that never ran. Silent failures, not loud ones, are what make governance tooling untrustworthy, so we kept the measurements that found nothing.

What's next for Ariadne

Right now we are targeting complaince to very important laws that target AI-assisted processes. Next, nightly unattended runs, more state-years so drift is real, S3 for artifacts, and the Incidents SDK upstream so findings can sit on the model rather than on its feature table.

Built With

• amazon-web-services
• caddy
• data-lineage
• datahub
• docker
• fastapi
• html
• langchain
• mcp
• mlflow
• opensearch
• pandas
• postgresql
• python
• scikit-learn
• sql
• yaml