Inspiration
Every ML model in production is carrying silent weight, a threshold changed under pressure three months ago, a feature added by an engineer who's since left, a preprocessing step nobody remembers the reason for. By the time something breaks, the first hour is always the same: reconstructing context that should have been written down but wasn't.
We kept asking the same question that no existing tool answers: of everything we're running right now, which model is most likely to fail next, and why do we actually believe that? DataHub already has the answer buried inside it, every model's full history of lineage changes, ownership, and governance metadata. What was missing was a way to check whether a given shape of change has, empirically, preceded a real failure before. Varve is that missing layer.
The name comes from geology. A varve is a single annual layer of lakebed sediment, thin, distinct, and readable, because it was laid down under specific conditions that no longer exist. ML models accumulate the same kind of layering. Varve reads those layers.
What it does
Varve ranks every production ML model by likelihood of near-term failure, backed by whether that exact debt pattern has caused a real incident in your organization before, not a generic industry heuristic, your actual history.
It reads DataHub's lineage graph archaeologically , not just what a model's dependencies are today, but when each dependency changed, who changed it, and whether the change was documented. It uses DataHub's Agent Context Kit to traverse multi-hop ML lineage (training data → feature store → model → deployment) and enriches every finding with real governance metadata and structured properties read directly from the graph.
It correlates across models, not just within one. A threshold change on Model A, made by an engineer who also touched Model B before leaving, and Model B failed 60 days later. No per-model diff can see that. Only a cross-model, actor-level join can.
It doesn't cry wolf, and it can prove it. A pattern that looks dangerous but has zero incident precedent gets explicitly downgraded, live, in front of you. Every decision is written to an append-only, SHA-256 hash-chained audit ledger, independently verifiable at any time.
It gets useful on day one, and smarter every day after. A brand-new team with no incident history gets an honestly-labeled industry baseline instead of silence. Every confirmed finding automatically upgrades that pattern to real, organization-specific evidence permanently, without any separate data-entry step.
It writes back to DataHub, not just to its own database. Every finding lands as structured metadata on the actual lineage node via both InstitutionalMemory annotations and a proposed new aspect type, ValidatedRiskPattern, so the next engineer or agent inherits the knowledge automatically.
How we built it
DataHub integration: We use DataHub OSS as the primary data source, reading lineage via the Agent Context Kit's get_lineage MCP tool (5-hop traversal across Postgres, S3, Snowflake, dbt, and Spark), ownership via OwnershipClass aspects with a documented priority resolution order, and governance tags via GlobalTagsClass aspects plus a conservative semantic inference layer for cold-start scenarios. Write-back uses DataHub's REST emitter to push both InstitutionalMemoryClass annotations and a structured ValidatedRiskPattern aspect proposal back to every affected lineage node.
Correlation engine: A 5-tier trust hierarchy resolves evidence in strict precedence, per-model incident history, per-actor cross-model history, org-wide patterns, and an industry-general baseline for cold-start, with explicit, visible evidence tier labels on every finding. The model never decides severity: a deterministic SQL join against the organization's own incident history makes every validity decision, and the LLM's only job is turning that decision into a clear, specific sentence.
Audit ledger: Every finding creation, severity resolution, downgrade, ownership routing, governance tag adjustment, write-back, and candidate incident confirmation is written to an append-only SHA-256 hash-chained PostgreSQL table. verify_ledger.py walks the full chain and confirms zero tampering, 38/38 entries verified in the current build.
Self-bootstrapping loop: A rolling Z-score anomaly detector on tracked business metrics proposes candidate incidents for human review. Confirmation writes a permanent row into the organization's incident history and immediately recomputes the pattern rollups, flipping evidence from industry-general to org-validated automatically, without any separate data-entry workflow.
Stack: FastAPI (Python), PostgreSQL 16, Next.js 16 + React 19, DeepSeek v4 Flash, DataHub Agent Context Kit, acryl-datahub SDK, Slack Block Kit for alerts, SHA-256 hash chain for the audit ledger.
Challenges we ran into
The actor/owner identity gap. DataHub's ownership metadata uses structured URNs (urn:li:corpuser:jonny1). Our lineage events carried free-text actor names (J. Alvarez). Nothing connected them. We built a 3-layer resolution engine, database lookup, initial+lastname matching, and a documented fallback with an explicit match-type label, to bridge the two identity spaces without silently inventing mappings that didn't exist.
Honest financial estimates without unconstrained projections. When we added financial risk estimates, early versions produced projections larger than the total monthly value of the affected dataset. We built a 4-tier baseline resolution hierarchy (DataHub native properties → DataHub-derived calculation → empirical median from business metrics → industry classification default) with a 100% tautological cap: Varve will never claim a projected loss larger than the total baseline value of the thing at risk. A post-generation sanitizer re-prompts the LLM if any dollar figure in the narrative exceeds the bounded baseline.
Making the fallback tier genuinely evidentiary, not a rubber stamp. An early version of the industry-general fallback silently passed through the provisional severity unchanged, a high-risk pattern type would stay HIGH regardless of the actual industry base rate behind it. We caught this ourselves and rebuilt the fallback to apply real threshold-driven severity caps (≥25% base rate retains severity, 15–25% caps at MEDIUM, <15% downgrades to LOW), then verified it independently across all three bands. The system now actively constrains the output at every evidence tier, not just at the org-validated one.
Accomplishments that we're proud of
- 6/6 ground-truth scenarios classified correctly - including a correct live downgrade from provisional-high to validated-low, with zero false positives, verified by an automated benchmark harness, not just a demo.
- 38/38 audit ledger entries cryptographically verified - with a deliberate corruption test proving the verification actually catches tampering rather than always printing success.
- DataHub write-back independently confirmed - read back directly from DataHub GMS after every write, proven idempotent under repeated runs against the same node.
- The cross-model actor history screen - the single thing no per-model tool can produce: a finding on one model, visible only because of what happened to a completely different model six months earlier, connected by a shared actor who has since left the team.
- A genuine open-source contribution - a ValidatedRiskPattern aspect RFC submitted to the DataHub repository (issue #18981), proposing a standardized, multi-agent-compatible way to write evidence-backed risk history back to the DataHub graph, something no agent can currently do in a queryable, structured form.
What we learned
Constraining where the model is allowed to act made the system easier to trust, not harder to build. Every finding's validity and severity are answers you can check by hand, the LLM is only ever asked to do the one thing language models are actually good at: write a clear sentence from a clear fact.
The valuable signal was never in any single table alone. Lineage events aren't new information, DataHub already has them. Incidents aren't new information, most teams already track them somewhere. The join between the two, at the right scope (per-model, per-actor, org-wide), is what turns "this looks undocumented" into "this exact pattern has cost this team money before."
We also learned, catching it ourselves before it reached a demo, that a fallback is only honest if it's actually evaluated, not just inherited. An early version of the industry-general tier silently rubber-stamped provisional guesses. Fixing that, making the fallback tier subject to the same evidentiary thresholds as validated data, turned out to be one of the most important corrections in the build, and it's now one of the system's independently verified behaviors.
What's next for Varve
- ValidatedRiskPattern as a merged DataHub aspect - move the RFC from proposal to an actual upstream contribution, so any agent can read and write structured risk evidence to the DataHub graph without inventing their own convention.
- A Watch Agent - background polling of DataHub for new lineage changes, auto-triaging high-severity hits and dispatching Slack alerts without requiring manual
/connectsessions. - Pattern severity learned from organizational outcomes - replacing hand-tuned thresholds with weights derived from an organization's own accumulated incident confirmations over time.
- Varve as a DataHub Skill - wrapping Varve's risk-ranking and finding-explanation capabilities as first-class DataHub Skills, callable from any Skills-enabled agent or MCP client.
- Cross-source correlation - extending the cross-model actor join to shared upstream data sources, not just shared human actors.
Log in or sign up for Devpost to join the conversation.