Silent Lucidity

A fare-prediction model's error metric creeps up over three days. No alerts. No failed jobs. Dashboard's green. This is a silent failure — the kind that costs money for weeks before anyone notices.

Comment

Silent Lucidity — Track 3, Production ML Agents

Inspiration

Every ML team has a story about the model that didn't break.

The job didn't fail. The pipeline didn't error. Every dashboard stayed green. And for three days, the model quietly served predictions built on data that stopped arriving — until someone finally looked at the numbers and asked why they'd stopped making sense.

We wanted to understand why that class of failure is so hard to catch, and the answer turned out to be sharper than we expected:

A catalog records that the ingestion ran. It does not record that the data arrived.

When an upstream load stalls, it doesn't raise an error — it just stops. Every table keeps reporting a healthy ingestion timestamp, including the one that has been frozen for three days. So the failure is invisible from the metadata alone, and the model has no way to tell "yesterday's data" from "today's data that happens to look exactly like yesterday's."

Meanwhile the thing that could answer the question — which upstream table actually broke — is sitting right there in DataHub's lineage graph. Nobody was walking it.

What it does

Silent Lucidity is an on-call agent for production ML.

When a model's error metric regresses with no accompanying failure, it starts at the model and walks DataHub's end-to-end ML lineage upstream through the MCP Server:

fare-predictor (mlModel)
  └─ Consumes ────▶ avg_fare_calibration (mlFeature)
       └─ DerivedFrom ────▶ mart_daily_summary, staging_trips
            └─ upstream ────▶ raw_trips

At every dataset hop it probes freshness in the data plane — because the catalog can't see this. The culprit is the first stale hop whose own upstream is still fresh. That boundary is the broken link.

Then it does three things most detectors don't:

It writes the answer back into DataHub. The incident report and causal chain are attached to the model entity. The culprit is tagged, with a dated root-cause note in its description. The next person — or the next agent — inherits all of it.

It installs the guard that would have caught it. A freshness assertion on the culprit, firing red in the Validation tab with its evidence attached (hours_since_last_load: 72).

And it knows when not to blame the pipeline.

That last one is the heart of the project. Our demo contains two incidents. Both trip the same alarm at the same magnitude — 14.2% model error. The verdicts are opposite:

2026-07-09 2026-06-27
Every upstream hop staging_trips 3 days stale all fresh
Verdict pipeline_failure real_world_anomaly
Action tag, report, install guard exonerate the pipeline

On 06-27 a storm cut trip volume in half. The data arrived on time — it was just unusual, and the dip propagated faithfully all the way downstream. The agent, in its own words:

"raw_trips is the root of the lineage chain (no upstreams) — the volume dip originates at the source, not in any pipeline stage."

A false "the pipeline broke" is worse than no diagnosis: it sends an engineer to fix a healthy system while the real cause goes unexamined. The error metric alone cannot tell these two apart. The context graph is the only thing that can.

Finally, the replay proves prevention isn't a claim:

date          staging lag      GUARD    model error   model alone would say
2026-07-07           24h     🔴 FAIL          7.89%   nothing wrong
2026-07-08           48h     🔴 FAIL          9.13%   hmm, elevated
2026-07-09           72h     🔴 FAIL         14.25%   CRISIS

The guard fires on the first missed load, while a model-only monitor still reads "nothing wrong" — two full days before the error reaches crisis.

Detection isn't the achievement. A 14% error gets noticed eventually, painfully. Firing before the model degrades is.

How we built it

The architecture is simple, and one design constraint shaped all of it.

An agent that pages a human must give the same answer twice.

You cannot put a diagnostic agent in a production incident path if it names staging_trips on Monday and mart_daily_summary on Tuesday from identical evidence. Reproducibility isn't a nice-to-have here — it's the difference between a demo and something on-call would actually trust. And it is genuinely hard, because the reasoning engine in the middle is stochastic by construction.

Our first instinct was the usual one: pin temperature = 0. That's the wrong answer, and on several current models it isn't even an available one — the parameter is rejected outright. More importantly it treats the symptom. Sampling settings make an LLM repeat itself; they don't make it right.

So we split the agent in two, along the line where determinism actually matters:

  • Evidence is deterministic by construction. Lineage traversal returns the same graph every time. MAX(trip_date) returns the same date every time. The agent chooses which hops to probe, but every fact it reasons over is a reproducible query result — never a recollection, never an inference.
  • The verdict is a strict-schema tool call, not prose. submit_diagnosis has a closed enum (pipeline_failure | real_world_anomaly) plus culprit_urn, lag_days, confidence, evidence[]. There is no free-text path to a conclusion. The model's explanation is allowed to vary between runs; the verdict fields are not.

We tested it rather than asserting it: 6/6 runs across both incidents — identical verdict, identical culprit, identical lag. The prose differed every time, exactly as designed.

Everything the agent reads and writes then goes through the DataHub MCP Server — get_lineage and get_entities to traverse; save_document, add_tags and update_description to write the verdict back. The freshness guard is emitted as assertion aspects, with Sentinel supplying the evaluation itself.

We registered the full Track 3 path in DataHub — mlModel → mlFeature → datasets → mlModelDeployment — so the agent walks genuine ML lineage rather than table lineage with a model bolted on top.

The one thing MCP cannot give us is freshness. So the agent carries a data-plane SQL probe alongside it — and this is the design, not a workaround:

Lineage tells the agent where to look. The data plane tells it what's wrong. The contradiction between them is the silent failure.

Two systems of knowledge, deliberately kept apart, and the gap between them is the diagnosis. That's the whole idea, and everything else is plumbing.

Runtime: a full incident is 4–6 agent turns, roughly 90 seconds and a few cents of inference. Cheap enough to run on every regression, which is the point.

Challenges we ran into

We built the demo, ran it, and the failure was invisible. The stall was right there in the data — and the model's error barely moved. Post-stall error was statistically indistinguishable from baseline noise. We rebuilt the harness with a different design and failed again.

The bug was in our thinking, not our code: a frozen feature costs nothing on a stationary timeline. If the world isn't moving, yesterday's data is a perfectly good substitute for today's. Staleness only costs money when the world drifts while your features freeze — which is exactly what happens in production, and exactly what a static test fixture doesn't do.

So we extended DataHub's datapack into a continuous 60-day window with drifting fares, re-running its own staging→mart transforms. The failure finally became expensive, the way it is in real life. It also gave us the honest version of our own claim: this is why silent staleness costs money — a sentence we could not have written before we watched it fail to.

Three writes silently did nothing. Our first writeback printed five cheerful success lines, and three of the four writes hadn't happened. MCP reports tool failures as result content (isError), not exceptions — an unchecked call_tool "succeeds" while writing nothing. We only caught it because we read DataHub back instead of trusting our own output. Every MCP call now checks.

The agent poisoned its own evidence. This is the one that nearly shipped. The agent writes conclusions into the catalog — and it also reads the catalog. Our leftover annotations from an already-fixed incident convinced it that a perfectly healthy pipeline was broken. It even noted every table probed fresh, then talked itself into a stall on the strength of a stale note we'd left behind.

This is the sharp edge of "write results back so the next agent inherits the knowledge": inherited knowledge is also inherited error. The agent now has an explicit evidence hierarchy — catalog annotations are prior claims, live probes are ground truth for this incident — and every note it writes is dated. We verified it adversarially by re-planting the misleading annotations; it now holds the correct verdict and calls the contradiction out loud.

A mis-built URN doesn't 404 — it creates a phantom entity. We wrote tags to a dataset URN missing its schema segment. The reads correctly failed; the writes silently succeeded, auto-creating a phantom table with our tags on it, sitting next to the real one. A demo showing tags on a phantom table would have been a silent failure of our own making.

The guard slept through the first missed day. Our freshness check used hours > 24. A table loaded daily that is exactly 24h stale has already missed a load — so a strict > doesn't fire until hour 25, forfeiting a third of the warning time on a three-day stall. One character (>=) was most of the point.

Accomplishments that we're proud of

The agent finds a bug it didn't plant, in a database it didn't build. The obvious objection to any demo like this is "you injected the failure yourself." So we pointed the agent at DataHub's nyc-taxi datapack exactly as shipped — no generator, no injected failure — and asked it to find the staleness DataHub planted there. It did: culprit staging_trips, 9 days behind, matching ground truth exactly. It needed no code changes to do it.

Better still, it didn't blame mart_daily_summary — which is also 9 days stale, and is the table a consumer actually reads. It called it "a symptom not the cause" and kept walking. That's the difference between reading a metric and reading a graph.

The exoneration case. Building an agent that finds a culprit is straightforward. Building one that looks at a 14% error and says "the pipeline is healthy, don't touch it" — and is right — required the context graph to be genuinely load-bearing.

The whole thing runs in one command. make demo takes a judge from an empty clone to detect → diagnose → writeback → replay, with the DataHub UI showing what the agent left behind.

What we learned

Metadata is not evidence. The most useful thing DataHub gave us was the map, not the readings. Lineage told us where to look with total reliability; freshness had to come from the data. Building an agent that treats those as different kinds of knowledge is what made it work.

Agents that write to the catalog they read from need a memory hygiene model. This surprised us, and we think it generalizes far beyond our project. The moment an agent's output becomes the environment's state, you have a feedback loop — and undated conclusions are landmines for whoever reads next, human or machine. Date everything. Rank live evidence above inherited claims.

Prevention is a timing argument, not a feature. Everyone will detect the 14% error. The interesting question is how early, and at which table — and that question is only answerable with lineage.

Read the API back. Between the phantom URNs and the silent MCP write failures, almost every serious bug in this project was something that reported success and did nothing.

What's next for Silent Lucidity

Contribute the workflow upstream. We've written a datahub-silent-failure skill for datahub-project/datahub-skills — the root-cause workflow, plus the OSS/Cloud guard reference and every trap above, verified against OSS quickstart. datahub-lineage tells you where to look and datahub-quality tells you what to install; nothing joined them for the silent-failure case.

Two small fixes to send back to the nyc-taxi datapack. We leaned on it hard, and found two things worth reporting: the failed-load day its README advertises (trip_count = 0) isn't present in the shipped .db, and the "3-day lag" lands as a single missing load across a 9-day calendar gap because the 2016 tail is sparse. Both are a few lines in create_db.py. The datapack did its job for us — the staleness it plants is real, and our agent found it — and these would make it sharper for the next team.

More failure classes. Freshness is one silent failure. Schema drift, volume collapse, and distribution shift are others, and the same lineage-walk-plus-probe skeleton handles all of them — the healthcare datapack (planted negative billing, NULL names, impossible dates) is the obvious next target.

Column-level lineage. DataHub has it. Today we name the stale table; we could be naming the stale column, and the feature that reads it.

Real warehouses. The data-plane probe is one function. Point it at Snowflake or BigQuery and the rest of the agent is unchanged.

Built With

  • acryl-datahub-(sdk-1.6.0)
  • apache-arvo
  • apache-kafka
  • as-the-openai-compatible-client)
  • datahub
  • datahub-mcp-server
  • docker/docker-compose
  • glm-5.2
  • gnu-make
  • graphql
  • mcp
  • ml-lineage
  • mysql
  • openai-python-sdk-(2.45.0
  • openrouter
  • opensearch
  • sqlalchemy
  • stdio-transport)
  • uv
Share this project:

Updates