Proveracus

An agent that watches the world's data feeds and news for emerging risks, ranks them by world impact, and uses DataHub's graph to find the hidden connection no single feed could.

Comment

Inspiration

I kept meeting the same problem wearing different clothes. A team renames a column and finds out weeks later that three dashboards have been quietly wrong. A drug's side-effect reports pile up in a public database long before anyone writes a story about it. A food recall and a drug recall turn out to share a cause, but they live in different systems and nobody connects them. The shape is always the same: something breaks upstream, and the damage spreads downstream where no one is looking. What was missing was never the data. It was one place that holds all of it and knows how the pieces connect. That place is a metadata catalog, and DataHub was the one worth building on.

What it does

Proveracus does two jobs, and they meet in the same catalog.

The first is a risk radar aimed at the outside world. It reads live public feeds, drug adverse events, food recalls, software vulnerabilities, aviation directives, alongside the day's news, and watches for the ones climbing abnormally against their own history. Everything, a quiet data spike or a loud headline, is judged on one question: how much should the world know about this right now. Each finding is written into DataHub as a small graph of the signal, the concept behind it, and its root cause. Proveracus then reads that graph back and looks for a cause sitting under more than one signal, which is how it surfaces a connection no single feed could show.

The second job turns that same analysis on your own data. You bring a dbt project, Proveracus reads it, shows you what is broken, and lets you fix it in plain English. Say "rename customer id to customer key" and it makes the change everywhere it belongs, through the models, the tests, and the BI dashboards DataHub can see even when dbt cannot. And because your project now lives in the catalog, it is not sealed off from the first job. The columns and tables you bring in enter the same risk analysis, checked for the silent leakage and drift that break a model in production long after it ships.

The reason these are one tool and not two is DataHub. Your tables and the world's signals land in the same graph, and the same engine walks both.

How we built it

DataHub is the foundation, not a bolt-on. Proveracus writes entities and lineage through the REST emitter and reads them back three ways: the native relationship graph, the raw upstream-lineage aspects inverted by hand so it works even before the index has warmed up, and the official DataHub MCP server. The core is a breadth-first walk over that lineage, ranked by Personalized PageRank so a node reached through many paths outranks one reached weakly. The detectors are Poisson EWMA and standardized CUSUM control charts, the same ones used in disease surveillance, so a real signal is a statistical decision rather than a threshold I picked. The agents that coordinate detection, root cause, and self-healing are LangGraph supervisors sitting over deterministic, tested engines. Model calls run through a provider-agnostic chain that falls back to grounded deterministic logic everywhere, so a missing key never takes the app down. It ships as a container, runs live on Render, and carries 397 tests, live integration ones included, that skip cleanly offline.

Challenges we ran into

The lineage index went dark on me mid-build. The real culprit was an OpenSearch out-of-memory crash, and the fix taught me to read lineage back from the raw upstream aspects so a query still works while the graph index is rebuilding. Cross-domain discovery was the hardest thing to make honest: a drug reaction and a food hazard are written in different vocabularies, so I map each raw cause onto a shared concept, with a guard that stops a coincidental match from inventing a connection that is not there. Detection had to earn its keep, so I score it with a backtest that reports lift, how much more often flagged things actually went bad, and a signal that adds nothing says so plainly. And renaming a column safely meant tracing it through CTEs and SELECT *, then reaching past dbt to patch the dashboards only DataHub knows about.

Accomplishments that we're proud of

The thing I am proudest of is that DataHub is genuinely load-bearing here, not decoration. Proveracus writes its findings into the catalog and reads them back to make its next decision, and there is one case it can answer that no single API can: a retracted paper reaching a live production service across four separate platforms. That only works because the lineage lives in one graph.

A close second is making the hard parts honest instead of just impressive. Cross-domain discovery could easily fake a connection, so it links a drug reaction to a food hazard only when they normalize to the same underlying concept, with a guard against coincidental matches. Detection uses the same control charts disease surveillance relies on, and every signal is scored by a backtest that reports how much more often flagged things actually went bad, so a useless signal is allowed to say so.

And I shipped it. The column rename reaches past dbt to patch the BI dashboards only DataHub can see, the whole thing runs live in a container, and it carries 397 tests, including live integration ones that skip cleanly offline. Doing all of that solo, and keeping it truthful about what it does and does not know, is what I am proudest of.

What we learned

The catalog is not where you file results. It is where the answer is found. The case that convinced me was a retraction reaching a production service across four separate platforms: impossible for any single API to assemble, and almost trivial once the lineage sits in one graph.

What's next for Proveracus

Widen the cross-domain concept map so more fields can connect on a shared cause, point the risk radar at a persistent DataHub Cloud instance so the hosted graph stays live between sessions, and turn the self-healing loop loose on a real estate so an upstream schema change proposes and verifies its own fix.

Built With

Share this project:

Updates