On-Call Data Engineer Agent

An agent that takes the 2 a.m. page: it walks DataHub lineage to the root cause, ranks the blast radius, files the incident, and writes a post-mortem back so the next incident resolves faster.

Comment

Inspiration

Every data team has an on-call rotation, and everyone dreads it. A dashboard goes wrong at two in the morning and whoever gets paged spends hours doing the same mechanical work: open the catalog, walk lineage upward table by table, check freshness and row counts at every hop, guess who downstream is affected, message some owners. If any energy is left, they write something down that nobody will ever find again.

That last part is the real loss. The knowledge from an incident evaporates, and the next person to hit the same broken pipeline starts from zero.

DataHub already holds everything needed to do this job properly: column-level lineage, assertions, freshness signals, ownership, usage statistics. The hackathon brief asks for agents that read DataHub, take action, and write results back to preserve knowledge for the next user or agent. I took that sentence literally and built the compounding memory loop it describes.

What it does

When a data quality assertion fails or a freshness SLA breaks, the agent triages the incident end to end:

  1. Recall. It first searches DataHub for post-mortems it wrote before, indexed as a searchable structured property on the datasets themselves. Memory runs before any traversal.
  2. Root cause. It walks column-level lineage upstream one hop at a time, asking at every table what a human would ask: is the assertion passing, is the data fresh, did the row count collapse, did the schema change. An explicit stop rule keeps it honest: a node is the root cause only when it is unhealthy and none of its own upstreams is. A table failing because its parent is stale is a symptom, not a cause.
  3. Blast radius. It walks downstream, collecting affected datasets, charts, dashboards and ML models, ranked by real usage statistics from the catalog.
  4. Act. It files a DataHub incident, tags the root cause down to the specific column, tags every impacted asset, and notifies the owners it read off the ownership metadata.
  5. Learn. It writes a structured post-mortem back into DataHub, so the next incident on the same lineage short-circuits.

A live web UI streams the reasoning as it happens: the tool-call timeline, the lineage graph lighting up node by node as the causal path resolves, the usage-ranked impact list, and a growing library of the agent's own memory.

The memory loop, measured

The point of the write-back is that the agent demonstrably gets better. The demo video shows two runs against the same root cause but different symptoms. The cold run was triggered by a row-count failure on agg_daily_rides. The repeat was triggered on agg_zone_demand, a dataset the agent had never triaged, whose root cause sits three lineage hops upstream. Between the runs the warehouse was healed while the post-mortem was deliberately kept, so the only difference was what the agent remembered.

Run Recall Time to root cause Tool calls
Cold no prior memory 83.0 s 92
Repeat prior post-mortem recalled and verified 71.5 s 70

That is 24% fewer tool calls and 14% less time, the same three hops, and the same correct answer. The repeat run still verified the recalled hypothesis against live evidence instead of trusting memory; if verification fails, it says so and falls back to the full walk. The verbatim /api/compare response for this exact pair is committed at examples/snapshots/compare-video-pair.json, and a second fully-committed pair (post-mortems, event logs, incident payloads under examples/) shows an even larger 62% time saving. The agent is not deterministic and wall clock moves between runs; the direction of the effect is what reproduces.

How I built it

Two DataHub surfaces, split deliberately. Catalog reads go through the DataHub MCP Server (six tools: search, get_entities, get_lineage, get_lineage_paths_between, list_schema_fields, get_dataset_queries). Seventeen more tools on the DataHub Python SDK and GraphQL cover what OSS MCP cannot reach, and that division is not reads versus writes: ten of the seventeen are reads (assertion status, freshness, row-count history, usage), five are writes, two are bookkeeping. On OSS, MCP's get_dataset_assertions is Cloud-only, so the trigger feed itself is derived from the Dataset.health field natively.

What it writes back. An incident via raiseIncident, dataset and column-level tags, an institutional-memory link, a searchable oncall.postmortem structured property (the recall index), a Document entity carrying the full narrative, and merged custom properties. Six surfaces, because each is discoverable in a different part of the DataHub UI.

Stack. OpenAI Agents SDK with a streaming run loop mapped onto a structured event contract, FastAPI with SSE (gap-free replay and reconnect), SQLite for run persistence, and a React 19 + Vite frontend with an xyflow lineage canvas.

Reproducible from a clone. A seeder builds the whole demo warehouse: 23 connected entities (15 datasets across raw, staging and marts, 4 charts, 3 dashboards, 1 ML model), 9 assertions, seeded queries, column-level lineage, ownership and usage, plus three breakage scenarios. Judges do not depend on any pre-loaded data. One script starts everything.

Challenges I ran into

Most of the hard work was discovering where DataHub OSS v1.5.0.6 behaves differently from its documentation. Five findings, each reproduced live and written up in the README for upstream:

  • operations(limit: n) truncates before ordering, so a small limit can silently drop the newest record. A dataset 26 hours stale read as 1.2 hours fresh.
  • The hasFailingAssertions search filter never populates; the trigger feed had to come from Dataset.health instead.
  • Two writes to the same timeseries aspect in one run can coalesce non-deterministically in the async REST sink.
  • add_lineage(column_lineage=True) silently writes nothing when two schemas share no column names. No error, no warning, just missing edges.
  • Asymmetric GraphQL fields fail the whole query, not the field: confidenceScore is writable but not readable.

The other challenge was making the agent discriminate. In the demo scenario the staging table is genuinely failing too, and a naive agent stops there. The intrinsic-versus-inherited stop rule is what makes it walk past the symptom to the cause, and a third scenario with a different root cause exists specifically to prove the answer is not hardcoded.

Accomplishments that I'm proud of

  • The measured memory loop: not a claim, a committed pair of runs with a compare endpoint.
  • Every number in the demo video is gated mechanically: the video build fails unless each on-screen figure appears verbatim in the filmed run's persisted log.
  • Write-backs that land where people already look: the post-mortem is one click from the root-cause dataset inside DataHub, not in a side database.
  • Honest limitations documented, including exactly which behaviors are Cloud-only and how the OSS workarounds behave.

What I learned

Verify every platform assumption against the running instance before designing on top of it. Nearly every expensive bug in this build was a silent wrong answer rather than an error: a truncated list, a coalesced write, a lineage edge that never got created. Errors are cheap. Silence is expensive.

And giving an agent a place to write down what it learned, inside a system other agents and humans already read, changes what the agent is. The memory is not a vector store bolted on the side. It is a structured property on the root-cause dataset, which means the next engineer finds it too.

What's next for On-Call Data Engineer Agent

Event-driven triggering through the DataHub Actions framework instead of polling. Confidence calibration that records whether a recalled post-mortem was ultimately confirmed and feeds that hit rate back into ranking. Of the five OSS findings above, the two reproducible ones are now filed upstream (https://github.com/datahub-project/datahub/issues/19063 and https://github.com/datahub-project/datahub/issues/19064), and the packaged datahub-incident-triage skill is now proposed upstream as https://github.com/datahub-project/datahub-skills/pull/141, with this project as its reference implementation.

Built With

Share this project:

Updates