Culprit: a stack trace for model decay

A stack trace for model decay. Give it a degraded model and a vague complaint, and it walks DataHub's ML lineage to the column that broke it, prices the damage in dollars, and opens the PR.

Comment

Inspiration

Model monitoring finds the cause inside the model. Data observability finds it inside the warehouse. Both do that well now. Neither one holds the fact that decides this case, because it sits on the boundary between them: which category values the deployed model was actually fitted on.

I wanted a real example rather than an invented one, so I went looking in the public NYC Taxi and Limousine Commission feed. In December 2024 a new taxi vendor started reporting trips under VendorID = 7. It arrived as 230 trips, about six thousandths of one percent of that month. By June 2025 it was 67,573.

Freshness, volume, null rate and schema checks all stayed green the whole time. The column kept its name, its integer type, its zero null rate and its normal row volume. The only visible trace at the source is one column's maximum value going from 6 to 7.

I ran the fuller sweep too, not just the checks that flatter the story, and four metrics do fire. Two are false alarms that predate the defect by months. Of the two that are real, unique_count on vendor_id goes 3 to 4 in December 2024, which is the same integer the max already shows and names no model. zero_count on the derived speed feature climbs, but in December 2024 that is 255 rows out of 3,502,209, against a baseline already swinging between 33 and 43. It does not become unmissable until March 2025, a quarter after the model started serving the new vendor wrong. And when it fires, what it says is that some speeds are zero.

That is the actual argument. Detection was never the hard part. Attribution is.

Meanwhile any fare model trained before December 2024 has a one-hot encoder written when only vendors 1, 2 and 6 existed. Every vendor-7 trip now tells that model it came from no vendor at all.

What it does

Culprit is handed a production model and a vague human complaint: quotes have drifted, nobody knows why. It walks DataHub's end-to-end ML lineage backwards from the model, through the feature table, through the dbt transforms, to the raw source column. It finds the change in meaning, proves that standard monitors would not have fired, measures the damage in dollars, and writes the finding back into DataHub so the next engineer or agent inherits the answer instead of rediscovering it.

On the real June 2025 data: $90,322 of attributable prediction error in a single month, across 66,146 real trips, while every standard data quality check stayed green.

There is a second defect stacked on the first, and I did not plant that either. Vendor 7 reports pickup and dropoff at the same second, so every one of its trips has a duration of exactly zero. The feature model guards against division by zero the way everyone does:

coalesce(trip_distance / nullif(trip_minutes / 60.0, 0), 0) as avg_speed_mph

The null-safety guard is what hides the corruption. Without it the column would have gone NULL and someone would have been paged. With it, the column stays clean and confidently reports 0 mph for 66,146 trips.

How I built it

The whole point was that nothing could be simulated, so the stack is real end to end.

19.3M real NYC TLC trip records across five months, loaded at true published volumes into DuckDB, no sampling anywhere. Real dbt transforms containing the actual defect. DataHub's native dbt connector parses the real manifest.json and catalog.json to produce genuine column-level fineGrainedLineage, so the dataset lineage is ingested build output rather than something I asserted.

No DataHub sample datapack ships ML entities, so the ML half of the graph is contributed by this project: 13 mlFeatures, an mlFeatureTable, an mlModelGroup, an mlModel and a dataProcessInstance training run, emitted through the DataHub Python SDK. Each feature records the source column it derives from, and that is what makes the walk from model back to column possible at all.

The agent reads through DataHub's own MCP server over stdio, which exposes 21 tools. Six are allowlisted into the investigation loop, and the mutation tools are deliberately held back for an explicit write-back step. Culprit does not reimplement catalog access.

A real scikit-learn model trained on 6.88M rows, plus a counterfactual control. Real write-back: an incident, a knowledge document holding the full trace, and an annotation appended to the offending source column.

Then the part I care about most. culprit fix locates the transformation at fault, patches it, runs dbt build against the real warehouse, and checks three gates before proposing anything: the build succeeds, the affected rows now match a category, and no other segment's row count changed. Only then does it open a PR.

The agent is genuinely uninstructed. Nothing about taxis, vendors or one-hot encoding appears anywhere in its system prompt. It gets a model URN, tools, and a method, and it works the problem.

On the dollar figure, which is the easiest thing in a project like this to inflate. Two models train with identical hyperparameters. The production model sees 2024-06 and 2024-09, so vendors 1, 2 and 6. The control sees those plus 2025-03, so its encoder knows vendor 7. Both score the full real June 2025 month. The obvious objection is that the control saw more data and fresher data, so some of its advantage is not the fix. That objection is correct, which is why the naive difference is not the headline. The control's unearned advantage is directly measurable on the segments that have no encoding defect: across 3,840,878 unaffected rows it is $0.0731 per row. Subtracting it gives a difference-in-differences estimate of $1.3655 per trip, which across 66,146 trips is $90,322.36. Both estimators are returned and the stricter one is the headline. The language model is never asked to produce a number; every dollar figure and row count is computed in SQL and handed to the agent as a fact.

Challenges I ran into

The volume monitor fired, and it was my fault. My first loader down-sampled the training months to 750k rows while loading the serving months in full. That made row volume swing 82%, which would have made a conventional volume monitor fire and would have destroyed the central claim of the whole project. The claim was being tested against an artifact of my loader rather than against the real feed. I reloaded every month at true published volume, moved sampling out of the warehouse entirely, and the swing dropped to 18%, which is ordinary seasonal variation.

My demo script had a number in it that was simply wrong. I wrote the voiceover before building, as a focus discipline, with placeholder figures. One of them claimed the model was overcharging by $2.14 a trip. When I actually measured the signed bias it was -$0.67 on vendor 7, close to vendor 2's -$0.80. The model is not systematically overcharging those trips at all. The damage is in error magnitude, not direction. The repo carries a correction log of every placeholder the data contradicted.

I published a finding that turned out to be wrong, and had to correct it in public. I reported that DataHub rejects an incident raised on an mlModel, here and in a comment on an upstream PR. When maintainers opened a follow-up issue asking which ML entities are affected, I probed a live instance instead of guessing, and the opposite is true: mlModel and mlFeature are accepted and the incident persists. What actually rejects is mlModelGroup, mlFeatureTable and dataProcessInstance. The real gap is that MLModel has no incidents field in the GraphQL schema, so an incident filed on a model is stored, correctly attributed, and impossible to read back. That is arguably worse than a clean rejection, because the caller gets an incident URN and no reason to think anything went wrong. I posted the correction on both threads with reproducible output and rewrote the finding in the repo. Another contributor then reproduced the rejection on v1.5.0.6 and flagged the contradiction, which exposed a second error of mine: my instance had been silently upgraded to v1.7.0 by a quickstart re-run, and I had been reporting the old tag for days. Both results are correct on the versions we each ran, so the behaviour changed somewhere between them. I found that by testing his exact URN first, since he used an unregistered data platform where I had used duckdb, and ruling that out left the version as the only difference.

DataHub SDK type surprises. MLModelProperties.version is a VersionTag record, not a string. Metric and hyperparameter values are strings, not numbers. MLModelGroupProperties does not accept a platform argument. Each of these raises an Avro exception that names the whole record and not the offending field.

Accomplishments that I'm proud of

The fix it refused to open. Culprit's first generated patch was where vendor_id in (1, 2, 6). That does not encode the new vendor. It deletes every one of its rows, 87,693 of them. It compiles. dbt build passes. The symptom disappears completely, along with the data. Anything checking whether the patch merely looks reasonable would have shipped it. Because the gate runs the patch against the real warehouse and compares row counts rather than reading the diff, it caught the deletion and refused to open the pull request. The patch it did open adds a catch-all bucket, so it will not break again on the next new vendor. That is the difference between an agent that writes plausible code and one that is allowed to touch a repository.

Finding a real incident instead of planting one. The fault in this repository genuinely happened in a public dataset, and anyone can verify it against the published feed in about a minute with python scripts/scan_tlc_semantics.py. VendorID = 7 is Helix, a real TLC-authorised provider, and the official data dictionary that says so is dated March 18 2025, three months after vendor 7 first appears in the data. The feed changed first and the documentation caught up later, which is the normal order of events and exactly the reason a catalog that records what a model was trained on is worth having.

Reporting the inconvenient finding. Vendor 6 shows a $1.387 attributable gap and an implausible 61.95 mph average speed. I did not plant it and did not know about it before building. It is worth only about $1,900 across 1,395 trips so it does not change the headline, but it is in the README, because suppressing a real finding would be worse than a slightly messier result.

Checking before filing upstream. I drafted a datahub-ml-lineage skill for the DataHub Skills registry, then found that another contributor had opened a PR two days earlier shipping a skill with the identical name and path. So I did not file it. The draft stays in the repo as working evidence. What I did file is the one finding nothing else covers: update_description with a column_path matching no field returns {"success": true} and writes nothing. An agent writing back to a catalog has no way to tell its write did nothing, so it reports success and moves on, which is a worse failure mode than an exception.

What I learned

The most expensive data defects are invisible to structural monitoring by construction. Freshness, volume, null rate and schema checks all watch the shape of data. When the meaning changes and the shape does not, there is nothing for them to see. Worse, the defensive coding practices that keep pipelines clean, like the nullif guard above, are often the very thing that hides the corruption.

The other thing I learned is narrower and more uncomfortable. Three times in this project I stated something confidently that measurement later contradicted: a dollar figure in my own demo script, a platform behaviour in a public GitHub thread, and then the version number I had been attributing that behaviour to. Every time, the fix was to go and measure rather than to reason about it, and the third one only surfaced because a stranger bothered to reproduce my result and tell me it did not hold. Every number in this README is now checked against the committed artifacts by a script that also fails the build if a retired claim reappears, which exists specifically because I could not trust myself not to reintroduce one.

What's next for Culprit: a stack trace for model decay

Unit changes and backfill-driven training leakage are described in the agent's method, but only the new-categorical-value path is exercised end to end here. The same traversal handles them and they need their own real examples, found the same way this one was rather than constructed.

The upstream gap is worth closing too. If MLModel gained an incidents field, the write half already works on v1.7.0, and an agent doing root cause on model degradation could leave the finding on the model itself instead of one hop upstream on a dataset that is perfectly healthy. Right now the model page reads clean while the model is the thing that is broken, and the model page is what an on-call engineer opens.

Built With

Share this project:

Updates