Unmeasured

4 stale alerts across two platforms collapse to 1 table to fix — and the datasets nobody declared an SLA for become visibly UNMEASURED in the DataHub UI, instead of silently absent from the report.

Comment

Inspiration

I run several production systems solo. The most expensive bugs I have shipped were never crashes — they were dashboards that kept looking right after the input went wrong.

Everyone idealizes the outcome; almost nobody writes down the worst failure first. Agents make this worse, not better: an agent infers from the nearest available result unless the structure forces it back to the source.

So I built the refusals before I built the answers.

What it does

Four stale alerts across two platforms collapse to ONE table to fix — and the datasets nobody declared an SLA for become visibly UNMEASURED in the DataHub UI, instead of silently absent from the report.

The full loop, in one pass: discover → measure → attribute → record → verify

Stage What it does What it refuses to do
discover Pages the catalog to exhaustion, then compares what it collected against the total the server declared Trust one page. Measured here: the catalog's search API declared total: 15 and returned 10
measure Puts the catalog's declared cadence and the data's actual MAX timestamp side by side — SQLite and Postgres, through one set of rules Emit a verdict with either number missing. That is UNMEASURED, with the reason attached
attribute Walks lineage to collapse N stale symptoms into root causes vs. downstream echoes — across the platform border Guess. Unreadable lineage is reported as unattributed, not as a root cause
record Writes every finding — including UNMEASURED — into the graph as tags + structured properties Invent vocabulary. Only 5 pre-declared terms are writable
verify Reads every write back and compares Trust the write response. A mismatch is reported as loudly as a failure

It also reads layered breaks: when an echo's data ends earlier than its stale source's, the transform stopped before the source ran dry. The official sample pipeline plants exactly this — a 9-day gap — and the agent says so instead of folding it into "stale because the parent is".

How I built it

Failure-path first, one commit each, in the order a failure would poison the next stage:

  1. UNMEASURED — before anything could report a number
  2. UNCOVERED — "no results" must never render as "no problems"
  3. STALE — only then, with both numbers printed side by side
  4. FRESH — a live-seeded control, because an agent that always says STALE would pass every test above
  5. discovery that must confess "walked N of M"
  6. attribution that refuses when lineage is unreadable
  7. write-back where a readback mismatch is a failure
  • Zero npm dependencies — built-in node:sqlite plus a bare stdio JSON-RPC client for the DataHub MCP server
  • 127 tests, no network needed — and mutation-verified: I broke each guard on purpose to confirm its own tests go red. Three of them did not. I only found that because I ran the check instead of repeating the claim — and those three are covered now.
  • The demo is a script in the repo (bin/demo.mjs --paced) — every beat is versioned, so what you watch is the script running for real, at the speed it ran. Two edits, both outside the agent's output: one window switch cut, one personal sidebar blurred. The recording also predates two committed output changes — a line printed while the MCP server starts, and no more "1 table to fix, not 1" when nothing collapsed. Same beats, same commands, same results; the repo has the newer script.

Challenges I ran into

Three findings changed the design — and two of them were my own bugs.

  1. The metadata clock is the wrong clock. The read-only MCP surface exposes no data-side update timestamp; I measured all five tools. And lastObserved would be wrong anyway — it is when DataHub last saw the metadata, not when data arrived. Using it produces a beautiful, wrong number. Recorded in code as a first-class UNMEASURED reason.
  2. My own acceptance test caught my own code on day one. get_entities returns not-found as an error payload inside an isError: false envelope. My presence check read an explicit "not found" as "found". The nearest signal beat the source — the exact failure this agent exists to prevent, committed by its author.
  3. It happened again, in my own filter. The day the catalog held both official instances, this repo's namePrefix filter silently swallowed the second one — nyc_taxi matched nyc_taxi_pipeline too, so a scan reported on a set that had quietly changed under it. The nearest match beat the actual identity, again. A filter bug is a coverage bug. Fixed, regression-tested, and kept in the README.

Accomplishments that I'm proud of

A closed loop that survives its own audit:

  • 6 datasets across 2 platforms in one scan
  • 4 STALE collapsed to 1 root cause + 3 echoes, with one echo on Postgres blaming a SQLite table
  • The planted 9-day mid-pipeline break read correctly, not smoothed over
  • Every write confirmed by independent readback; re-runs are idempotent, and human tags are never touched

Open-source contributions to DataHub

Building against the official sample dataset meant reading it the way this agent reads a catalog. That surfaced six reproducible problems, filed upstream as three issues plus a fix PR — all public:

  • datahub#18753 — a platform-level bug: ghost edges in the graph index. Three storage layers returned three different answers about the same write, hours after it landed. Reproduced 3/3 across three client versions, against a 0/10 control group. This is why the agent reads every write back instead of trusting the write response. Update, Aug 3: closed as completed — a DataHub maintainer thanked me for the analysis and traced the underlying cause to a fix landing in v1.6.x (datahub#17204), with an ELASTICSEARCH_THREAD_COUNT=1 workaround for v1.5.x. The agent never depended on the bug existing — readback is unconditional — so it behaves the same on patched clusters and on the v1.5.x deployments where the race remains.
  • static-assets#218 — three bugs in the sample dataset's own metadata script. The one that matters most here: it emits a whole aspect per tag, so each emit replaces the last and every table ends up with exactly one surviving tag — silently removing the very daily_refresh declaration a freshness agent reads. Also filed: ownership points at a group that is never created, and --dry-run prints a success banner while doing nothing.
  • static-assets#219 — two documentation-versus-data mismatches. The README describes staging stopping 3 days before raw; the actual set difference is a single missing load day, which reads as 9 calendar days because the dates are sparse — confirmed by row math (248,315 − 208,675 = 39,640, exactly that day's rows). That 9-day gap is the one this agent reports. The row count is documented as both 500k and 250k; the committed data is 250,000 exactly. I trusted those numbers only because I measured the data instead of reading the docs — which is the same rule the agent runs on.
  • static-assets PR#220 — the fixes for all three bugs in #218, +92/−33 in one file, each tested against a live quickstart. It also fixes the Windows console crash another participant reported in #211. Open, no conflicts, awaiting review. #219 is a documentation-versus-data mismatch and is not addressed by this PR — it needs a maintainer decision on whether to regenerate the planted variant or correct the README.

I also posted a public self-correction on #18753: my original report said the inconsistency persisted "2.5+ hours", which was an estimate I had not actually timed. I retracted it, replaced it with the documentable ~2h15m, and stated which parts of the report were unaffected and why. A bug report is a measurement claim like any other; if the agent has to say how it knows a number, so do I.

Status, stated plainly, as of August 8: #218, #219 and PR#220 remain open and unmerged. #18753 is closed as completed — that one now has a human maintainer's answer, and the account above reflects what they wrote, no more.

What I learned

Define the error you refuse to see, before the outcome you hope for. A "perfect result" is an essay with no exam question — anything can grade itself well against it.

And a write response is a claim, not evidence. Three storage layers gave three different answers about the same write. So this agent reads every write back — and the demo's finale asks the graph directly over GraphQL, a channel the agent does not own.

What's next

  • More readers behind the same boundary — BigQuery or Snowflake is the same exercise as Postgres was; the honesty rules live above the platform boundary and survive the swap untouched
  • A CI gate mode — the self-check gates already exit non-zero
  • Freshness history — consecutive runs writing trend, not just state
Share this project:

Updates