Inspiration
Everyone’s shipping AI agents and ML models, but the failures that hurt most are silent: a column disappears upstream, a feature table goes stale, ownership is missing, and the model keeps serving as if nothing changed. Observability tools often catch that after money is lost. We wanted a pre-deployment gate that uses DataHub’s context graph — schemas, lineage, ownership, quality, and governance — so agents can decide ALLOW vs BLOCK before a bad model hits production.
What it does
ModelGuard is a Production ML Agent that: Walks ML lineage from source datasets → feature sets → model → deployment Detects schema drift, stale data, quality regressions, missing owners, and unapproved PII-derived features Scores risk (0–100) and returns an auditable ALLOW / BLOCK decision Produces a DataHub write-back plan (tags, owners, incident evidence) so the next engineer or agent inherits the context Ships a judge-friendly dashboard plus sample risk reports in /examples Demo scenarios include a healthy fraud model and a “silent upstream failure” where amount vanishes from transactions and the deploy is blocked.
How we built it
FastAPI backend with typed models and a deterministic risk engine (policies first; LLM only for summarization later) DataHub-shaped lineage context (datasets, features, ML model, deployment edges) Responsive dashboard for lineage, findings, risk score, and write-back plan Tests, Dockerfile, and Render config for cloud demo Local DataHub Docker quickstart + showcase sample pack used during development Apache 2.0 public repo: https://github.com/RiyaTallsss/modelguard
Challenges we ran into
Balancing a real DataHub integration with a 21-day Cloud trial vs judging through Aug 31 — solved by local quickstart + a durable demo mode Keeping decisions auditable: risk must come from deterministic rules on lineage signals, not opaque LLM guesses Resource constraints running DataHub locally (~8GB RAM) alongside other Docker workloads Shipping something judges can run without private credentials while still targeting MCP/Agent Context Kit write-back
Accomplishments that we're proud of
An end-to-end vertical slice: scenario → lineage analysis → BLOCK/ALLOW → evidence → write-back plan A clear product story for the Production ML Agents track Clean repo structure: tests, license, examples, deploy configs, and a working UI Framing silent ML failure as a metadata-graph problem DataHub is uniquely positioned to solve
What we learned
Context beats chat: agents need lineage, owners, and quality — not just prompts Write-back matters as much as read: tagging ml-risk:high and saving evidence makes the catalog smarter for the next agent Deterministic gates + explainable findings are more trustworthy for deploy decisions than free-form AI alone Hackathon submissions win on demo clarity as much as architecture
What's next for ModelGuard
Live DataHub MCP / Agent Context Kit read + write-back (tags, docs, owners) Auto-open GitHub remediation PRs/issues on BLOCK Historical schema snapshot comparison Open-source DataHub skill: /ml-risk-audit Hosted public demo for judges without local Docker
Log in or sign up for Devpost to join the conversation.