Inspiration
I keep hearing the same story from people who work with ML in production: nothing breaks loudly. A model just quietly gets worse. Someone leaves the company and takes the only knowledge of how a model was trained with them. A feature table stops refreshing and nobody notices for a month because the dashboard is still green. By the time anyone catches it, the damage is already done and nobody can even say when it started.
What got me thinking about this specifically was realizing that DataHub already has all the information needed to catch these problems. It knows the full path from a raw table, to a feature, to a model, to a deployment. That felt like the obvious thing to build for this hackathon: not another monitoring dashboard for latency or uptime, but something that uses the one thing only a metadata graph can see.
What I learned
The biggest lesson was that the real API almost never matches the docs, and you should verify everything against a live instance before designing around it. I originally planned the whole client layer around assumed shapes for the MCP tools (things like a get_entities call returning properties.description). Almost none of that held up once I actually spun up a DataHub quickstart and called the tools for real. Fields were flat instead of nested, results came wrapped differently depending on the tool, and a few things I needed (like DataProcessInstance timestamps and the mlModelTrainingData aspect) simply weren't exposed through MCP at all and needed a raw GMS call instead.
I also learned that DataHub's write tools are disabled by default on a self hosted MCP server unless you explicitly set TOOLS_IS_MUTATION_ENABLED, which cost me a while to track down since the failure mode is just "the tool silently isn't there."
How I built it
The backend is FastAPI with a single /scan endpoint that streams findings over Server Sent Events as each check finds them, instead of making you wait for a full run. Each of the five checks (Skew Sentinel, Freshness and Leakage Auditor, Duplicate Feature Finder, Zombie Model Hunter, and Provenance Auditor) is a plain Python generator that takes a DataHub client and yields findings. They don't know FastAPI or the frontend exist at all, which made them straightforward to unit test with a fake in memory client instead of a live DataHub instance.
datahub_client.py is the only file allowed to actually talk to DataHub. It wraps the MCP tools for search, lineage, and entity lookups, and falls back to acryl_datahub's DataHubGraph SDK for the handful of things MCP doesn't expose, including the Timeline API that Skew Sentinel uses to compare a model's schema at training time against what's live now.
For the demo data, I wrote a seed script that populates a fresh DataHub instance with a small realistic scenario: an orphaned model, a model trained on PII tagged data, a stale feature table, a pair of duplicate features, and a dataset whose schema actually changes after a model was trained on it, using real wall clock timestamps so the Timeline API has genuine history to show.
Challenges I faced
The hardest one was Skew Sentinel. There is no MCP tool for historical schema versions, so I had to go find DataHub's Timeline API directly (it lives at /openapi/v2/timeline/v1/{urn}), figure out its filter and category syntax from the CLI source code since it isn't documented anywhere obvious, and then seed real schema history by emitting two versions of a dataset's schema a few seconds apart so there was actual drift to detect.
Log in or sign up for Devpost to join the conversation.