fray | Devpost

fray

Inspiration

What it does

How we built it

Challenges we ran into

Inspiration

Models do not usually die loudly. A column quietly changes type, a table stops landing, a pipeline gets rewired under the feature store, and the model keeps serving predictions that are slowly wrong. Nobody gets paged because nothing threw an error. Meanwhile all the information needed to catch it early is already sitting in DataHub, unused, because nobody wired the lineage to the thing that actually breaks.

What it does

fray compiles each production model's upstream contract by walking DataHub lineage backwards from the deployment through the model, its features, their source tables, and everything upstream of those. Fields and types, freshness, row counts, null rates, the lineage edges themselves. Nobody configures anything.

Then it rescans and diffs. Schema drift, freshness stalls, volume cliffs, null spikes, pipelines rewired under the contract. Every finding gets a blast radius, because fray inverts the lineage and scores each break by which models and which live deployments inherit it. On call reads that the fraud detector production endpoint is at risk, not that some table has nulls.

Then it writes everything back into DataHub. Custom assertions with failure run events and evidence on the broken tables. Success heartbeats on the clean ones. Tags for contract broken, at risk, and monitored. An institutional memory link on every at risk model. A markdown handoff brief saved as a DataHub Document through the Agent Context Kit. The graph is both the input and the output, so the next scan and the next agent start from what the last run learned.

There is an optional Claude layer that groups raw findings into incidents by probable root cause and writes the paging message. The deterministic engine does not need it.

How I built it

A small port interface sits between the detection engine and the graph, with two implementations. One talks to a live DataHub through the SDK and the Agent Context Kit. The other reads a recorded metadata snapshot and collects the writes in memory. Same engine, both paths.

That decision is why the demo needs no infrastructure. Clone, pip install, fray demo, and the whole thing runs on committed fixtures: a two model stack with four planted silent breaks, and a dump of the exact thirty metadata change proposals a live run publishes. The fixtures are generated from DataHub's own typed aspect classes, so they are valid metadata by construction, and the same file replays into a real instance.

Challenges I ran into

The Agent Context Kit is uninstallable under its own pin. Version 1.7.0 requires acryl-datahub 1.6.0.6 but imports a module that only exists in 1.7.0, so a clean install produces a package that cannot import, and any project depending on both hits a resolver conflict. I documented the workaround and reported it upstream as issue 19015.

The subtler one was scoping. When a pipeline gets rewired and a table falls out of the contract, a naive diff reports every column on that table as removed. Six loud findings for one actual event. It now reports the edge cut and a single informational note that the table left scope.

Accomplishments that I am proud of

The blast radius. Everything else is a detector, and their docs already ship a data quality agent example, so detection alone is table stakes. Ranking a null spike by which production endpoints inherit it is the part that changes what someone does at three in the morning.

What I learned

How much is already in the graph that nobody consumes. End to end ML lineage across deployments, models, features and datasets exists in DataHub as first class entities, and almost nothing reads it as a dependency contract. Also that writeback is the whole game for an agent. An agent that reads context and keeps its conclusions in a terminal buffer has not made the platform any smarter.

What is next for fray

Per column contracts derived from feature transformations rather than whole table snapshots, thresholds learned from profile history instead of fixed defaults, and opening an incident in DataHub when a break reaches a live deployment.

Upstream bug reported while building: https://github.com/datahub-project/datahub/issues/19015