Janus

janus is an ML deployment gatekeeper that reconstructs whether your training data was healthy at the moment you trained, and writes the evidence back into DataHub as a Model Passport.

Comment

Inspiration

Every data quality tool answers the same question: is this data healthy right now?

Nobody asks the question that actually matters for a model: was it healthy at the moment we trained on it?

Those are different questions, and the gap between them is where models quietly break. An ETL pipeline stalls at 2pm. Someone files an incident. A training job runs at 3pm on stale data. The pipeline is fixed and the incident is resolved at 5pm. By the time anyone reviews that model for deployment, every check is green — the dataset is fresh, the assertion passes, the incident is closed. The catalog shows no problem, because right now there is no problem.

But the model trained inside the window. The corruption is already baked into the weights, and it is invisible to every after-the-fact check, forever.

I kept coming back to the fact that DataHub already stores both halves of the answer. It records incidents with created and resolved timestamps. It records assertion runs over time. It records models, their training runs, and their upstream lineage. Both facts are sitting in the same graph — nobody had joined them on time.

That join is janus.

What it does

janus is a deployment gatekeeper for ML models. Point it at a model in DataHub and it reconstructs the health of that model's training data as of the moment it trained, then writes a signed evidence document — a Model Passport — back to DataHub.

For any model, it:

  1. Traces upstream lineage — walks model → features → feature tables → datasets to find every training data source
  2. Performs the temporal join — for each upstream dataset, asks whether an incident was active or an assertion was failing at the training timestamp, including ones since resolved
  3. Scans governance — flags PII, restricted, and deprecated data in the training pipeline
  4. Compares against the champion — finds the currently-deployed model and diffs the metrics
  5. Renders a verdict — APPROVED / BLOCKED / NEEDS_REVIEW, derived in code from configurable rules, not by the model
  6. Writes the passport back — into the model's customProperties, plus a verdict tag and a one-line description

The demo case, with real numbers

fraud-detector_6 trained at 15:26:16. The freshness incident on its upstream fraud_transactions was raised at 14:24:02 and resolved at 17:24:02 — the model trained 62 minutes inside a 3-hour outage window.

That incident is RESOLVED. DataHub's UI shows the dataset green. Every current-state quality check passes.

janus blocks the model anyway, on three independent grounds: the incident overlap, an email must not be null assertion that was FAILING on user_profiles at that same instant, and accuracy (−0.37%) and f1_score (−45.95%) regressions against champion fraud-detector_5.

And it discriminates. fraud-detector_5 trained twelve days earlier, on the same two datasets, with the same incident history sitting in the graph — and comes back with clean data_health on both. It isn't flagging anything that has ever had an incident. It's reconstructing a moment in time.

Both passports are in the repo as real captured output, not hand-written samples.

How I built it

Everything the verdict rests on comes out of DataHub. janus uses essentially every surface the platform exposes:

DataHub surface How janus uses it
Context graph Incidents with temporal bounds, assertion run history, ML lineage, governance tags — the entire evidence base
Agent Context Kit 8 pre-built LangChain read tools (get_lineage, get_entities, search, get_dataset_assertions, …)
MCP Server — consumed Optionally reads DataHub through mcp-server-datahub as a third independent read path
MCP Server — served janus is an MCP server: Cursor and Claude Code can ask "is this model safe to deploy?" before writing the deploy PR
Skills A datahub-ml-passport skill, matching the conventions of DataHub's own datahub-search / datahub-lineage skills
DataHub Actions Subscribes to MetadataChangeLog and evaluates a model automatically the moment it's deployed
Write-back Passport into customProperties, verdict tag, and description — all via GraphQL mutations

Six custom tools do what the pre-built ones can't: the temporal incident checker and assertion checker (the core innovation), a multi-hop lineage tracer (a single ACK get_lineage hop stops at the model, so it walks the mlFeature/mlFeatureTable hops itself), a governance scanner, a champion comparator, and the validated writer.

The LLM never decides the verdict. It gathers and structures evidence; rules.py derives the verdict deterministically from that evidence, so the same facts always produce the same answer. Teams override the defaults in configs/verdict_rules.yaml, and rules may only escalate — if the agent blocked a model for a reason the rules don't model, that block stands.

Four ways to consume it, because the moment of use differs by team: an Actions daemon (automatic, no human in the loop), a FastAPI endpoint for CI/CD, an MCP server for AI editors, and a CLI.

Runs on DeepSeek deepseek-v4-flash at roughly $0.004 per evaluation.

Challenges I ran into

The tests were passing on an event that doesn't exist. The Actions trigger listened for EntityChangeEvent on mlModelDeployment and had green unit tests. When I finally deployed it against live DataHub, it never fired. I drained the PlatformEvent_v1 topic and found DataHub OSS only publishes EntityChangeEvents for a fixed set of entities — dataset, container, tag, assertion, dataProcessInstance. No ML entity and no incident is ever published there. The handler was correct about an event that is never emitted; the tests passed because they asserted on hand-written payloads. Rewrote it onto MetadataChangeLog, which carries every aspect write, and proved it fires by emitting a deployment and watching a passport appear for a model that had none.

The gate failed open. The CI endpoint parsed the verdict out of the agent's closing message — which is prose, not JSON — so parsing always failed and it fell back to a permissive needs_review with no blocking reasons. A model blocked on three independent grounds came back as something a pipeline gating on blocked would happily deploy. It now reads the stored passport from DataHub and raises rather than ever guessing. Every existing test had mocked the very function that contained the bug.

Failing toward "approved" is the failure mode that matters. This kept recurring in different disguises: a GraphQL error swallowed in the incident tool, a hollow passport written with BLOCKED and empty evidence arrays, one bad tool argument aborting an entire evaluation. For a gatekeeper, silence and success look identical from the outside. Nearly every guard in the codebase exists because some path found a way to be quiet about a problem.

Timestamps are the whole ballgame. An unparseable timestamp defaulting to 0 would sit before every incident ever recorded, so every check would return "healthy" — a silent, total failure of the one thing janus does. It raises instead.

Accomplishments that I'm proud of

The temporal join works and discriminates — proven live on two models with identical lineage and opposite verdicts, separated only by when they trained.

It fails loud everywhere it counts. A gatekeeper that stays quiet when it breaks is worse than no gatekeeper, and I found three separate ways it could have.

All four consumption modes are proven end-to-end against live infrastructure, not just unit-tested — including the Actions trigger that had never once fired.

132 tests, and I verify new ones actually fail against the buggy code before trusting them.

What I learned

A passing test proves nothing about an event you've never observed. The Actions bug is the cleanest lesson I've had in a while: three green tests, correct logic, zero real-world firings. Now I drain the actual topic before trusting a subscription.

"Resolved" is a lie the catalog tells about the past. Current state is a terrible proxy for historical state, and every tool I looked at conflates them.

Deterministic beats clever. Moving the verdict out of the LLM and into rules made janus auditable, reproducible, and configurable in one change. The model is very good at gathering evidence and should not be trusted to grade it.

What's next for janus

  • Backfill mode — evaluate an entire estate historically to find models that have been carrying wrong passports for months
  • Richer refresh triggers — a new incident already re-evaluates affected models; assertion regressions should too
  • Approval thresholds — let teams express "PII is acceptable in this domain" so APPROVED is reachable where it should be
  • Contribute the Skill upstream to DataHub's skills repository

Built With

Share this project:

Updates