Model-Sentinel

Inspiration

Everyone's shipping agents that can call tools. The failure mode we kept seeing is quieter: a model stays IN_SERVICE after upstream schema drift, a failing quality check, or a deprecated feature path — and nothing in the serving stack complains until revenue or trust is already gone.

DataHub already stores the graph those failures live on — training data → features → training runs → models → deployments. The gap isn't another text-to-SQL demo. It's a closed loop: read that lineage, detect silent breakers, and write risk back so the next person or agent inherits the context.

That lined up cleanly with the Production ML Agents challenge — and deliberately away from rebuilding Analytics Agent or wrapping the shipped Skills registry.

What it does

Model Sentinel scans a production ML model's DataHub lineage and runs five deterministic detectors:

• schema drift (feature column missing upstream)
• upstream quality failures
• deprecated feature/source paths
• ownership gaps on prod-bound models / critical upstreams
• stale deployments (upstream changed after the last training run)

On findings it mutates the catalog (model-at-risk, sentinel-severity:*, description append) via Agent Context Kit tools, then emits a scan report and a metadata-aware Dagster retrain stub under examples/.

Judges can try it without Docker:

pip install -e ".[dev]"
model-sentinel scan --demo
uvicorn demo.app:app --port 8000

How we built it

• Protocol-first design: GraphPort with InMemoryGraphPort (planted fixture) and DataHubGraphPort (ACK: search, get_entities, get_lineage, add_tags, update_description).
• Deterministic core, optional LLM: scan works with zero API keys; investigate is an optional LangChain agent on the same tool surface.
• Contract tests that fake ACK tool I/O so lineage + write-back are proven without a live GMS.
• Full fixture ingest script for local DataHub quickstart when Docker is available.
• Stack: Python 3.12, Pydantic, Click, FastAPI, pytest, ruff, GitHub Actions, Apache-2.0.

Repo: https://github.com/tarang-tj/model-sentinel

Challenges

1. Originality under a strong platform. DataHub already ships powerful agent surfaces. We had to kill “thin Skills wrapper” and “Analytics Agent clone” early and stay in the ML-guardian gap.
2. Demo vs live truth. No Docker on the build machine meant the default path couldn't pretend to be live GMS. We shipped an honest dual mode: offline fixture for judges, ACK adapter + ingest for real instances, and contract tests that exercise the real tool call shapes.
3. Python / dependency reality. Live extras (acryl-datahub) needed ≤3.12 on our toolchain — so we pinned the judge path and documented it loudly instead of papering over wheel failures.
4. Write-back that judges can see. Reading lineage scores points; mutating the graph is what matches the hackathon brief. Tags + description blocks were non-negotiable, including in the demo UI.

What we learned

• For production ML risk, deterministic graph checks beat embedding retrieval — the breakers are structural (missing columns, tags, timestamps), not semantic similarity.
• “Agent” for this problem means tools + judgment + inheritance: the valuable part is leaving the catalog smarter than you found it.
• Submission quality is part of the product: examples/, a click-through UI, CI that runs scan --demo, and a honest README matter as much as the detector logic.

What's next

• Record the <3 min demo with live DataHub write-back visible in the UI.
• Upstream skills/datahub-ml-sentinel to the DataHub Skills registry.
• Wire real assertion aspects and richer deployment metadata on the live path.

Built With

• actions
• agent
• apache-2.0
• click
• context
• dagster
• datahub
• fastapi
• github
• jinja
• kit
• langchain
• mcp
• openai
• pydantic
• pytest
• python
• rich
• ruff
• uvicorn