Janus

Janus: the missing CI for your ML supply chain, built on DataHub.
Our documentation site (docs.ahmedxsaad.me): what Janus catches, how it works, backed by real numbers, narrated by Argos.
Not a suggestion. Janus writes the model-at-risk tag straight into DataHub's own model page.
The same DataHub lineage graph Janus reads to walk from a stale table to every model downstream of it.
GIF
Argos, the DataHub watchdog. Idle, but never off duty.
Argos flags a stale table the moment it crosses its SLA.
Argos walking the blast radius one lineage hop at a time, from a raw table to the feature it feeds.
Running as a systemd service on the same VM the live demo runs on, scanning every thirty seconds since before this screenshot was taken.
Real target leakage, caught before it merges. Exit code 1, same as any failing test.
Run the same scan twice and nothing duplicates: the incident is reused, not recreated.

Inspiration

Models fail quietly. A table goes stale, a label leaks into a feature, an owner leaves and nobody notices, and the first real sign of trouble is a metric drifting weeks later in production. DataHub already tracks lineage and ownership for every dataset and model in the catalog, but that information mostly sits there, passive. Nobody was reading it, correlating it, or acting on it in real time.

Janus exists to close that gap: turn the metadata DataHub already has into an active judgment about whether a specific model can be trusted right now, backed by evidence instead of a gut feeling.

The name comes from Janus, the Roman god of thresholds and transitions, traditionally shown with two faces looking in opposite directions at once. That felt like the right metaphor for what the tool actually does. It looks backward through a model's full upstream lineage to find what could be wrong, and forward through its blast radius to see who gets hurt if it is.

What I learned

Most of what I learned was about designing for a metadata graph rather than a normal database. Writing evidence back into DataHub on every scan sounds simple until you realize a scan that reruns every 30 seconds could spam the catalog with duplicate incidents forever if writes aren't keyed carefully. Idempotency had to be a first class design constraint, not an afterthought.

I also learned that a trust score only means something if it shows its work. An integer out of 100 invites a reader to compare it against a threshold nobody agreed on, so the score had to come last, after the deductions that produced it, each one naming the actual finding responsible:

$$\text{trust score} = \max\left(0,\ 100 - \sum_i w_i\right)$$

where each w_i is the weight of a specific finding, for example, upstream failure (w = 40), target leakage (w = 20), freshness lag(w = 15), or missing ownership (w = 10).

How I built it

Janus walks a model's full upstream lineage through DataHub's graph and runs it through a set of detectors: target leakage (a feature whose lineage traces back to the label it is supposed to predict), freshness lag (an upstream table that has gone stale against its SLA), missing ownership, and upstream pipeline failures. Each finding subtracts a fixed weight from a starting score of 100, and the model's final trust band comes out the other side.

Everything a scan finds gets written straight back into DataHub: an incident on the offending dataset with the exact column chain as evidence, a model-at-risk tag on every downstream model inside the blast radius, trust_score and trust_band as structured properties on the model itself, a guarding assertion on the source table, and a Model Impact Report linked from the model. All of it is keyed so a rerun that finds nothing new reuses the existing incident and tags instead of duplicating them.

Janus runs three ways. As a systemd service polling continuously in production, not a one-off script for a demo. As a CLI you can point at one model or the whole catalog. And as a CI gate that exits non-zero and blocks a merge the moment it finds something above a severity threshold, the same way a failing test would.

It also speaks MCP, so an agent can ask "can I trust this model" directly and get the score and the reasoning behind it back, instead of a human having to go look it up.

Challenges I ran into

Idempotency was the hardest problem in the whole project. It is easy to detect and write back once. It is much harder to make "detect and write back" safe to run every 30 seconds forever without turning the catalog into noise.

Getting the CI gate right was its own challenge: a gate that blocks too eagerly gets disabled by the first team it annoys, so the detectors had to be precise enough to earn the right to block a merge, not just flag a warning.

Walking a large lineage graph fast enough to run inside a CI pipeline, rather than timing out, also meant being deliberate about how much of the graph a scan actually needs to traverse versus what it can skip.