Inspiration
My day job is running building automation for a hospital — mission-critical infrastructure where "mostly works" isn't acceptable. When a chiller drifts, someone gets paged before the ward gets warm. Then I look at ML pipelines and see the opposite: the training data can silently shift under a serving model for weeks and nobody notices until the downstream metrics tank. The lineage graph is already sitting in DataHub. What's missing is something that stares at it on a schedule, catches the drift, and pages the on-call before the next deploy — with a root cause they can act on in thirty seconds. That's Ogle.
What it does
Ogle walks your DataHub lineage graph on a schedule and catches silent training-data drift before it reaches production. Given a deployed model, it traverses upstream through the lineage graph (model → feature tables → source tables) and, at each hop, computes a lightweight signature it compares against the last known-good state across nine drift dimensions — schema, volume, quality (null-rate), categorical distribution, freshness (stale-feed SLA), the per-feature numeric moments mean, stdev, and range, and distribution shape (a histogram reshaping while its mean and stdev both hold — the residual the moment rules miss).
Under the distribution/shape dimensions sits a distance layer that folds a whole-feature move into a single number: a parametric half (Hellinger, Jeffreys/PSI, 2-Wasserstein) modeled from the mean+stdev already in the signature, and an empirical half computed from raw quantiles — a bounded Jensen–Shannon gate corroborated by the Kolmogorov–Smirnov / Kuiper / Cramér-von Mises / Cramér-energy / Watson U² family and the W1 ≤ W2 ≤ W∞ transport ladder — catches multimodal or skew moves the Gaussian summaries idealize away.
When an above-threshold anomaly hits a serving-path asset, Ogle writes an actionable narrative — what changed, when, who owns it, which downstream models are exposed, direct link to inspect — and writes findings back into DataHub as tags on the affected assets, plus a persistent incident memory so the agent stops re-paging on incidents it has already reported. Runs as one-shot ogle check or a scheduled ogle watch loop that pages once per incident.
How we built it
Python agent, five composable stages: a 88scheduler88 (cron/APScheduler-friendly watch tick), a lineage-walk that traverses DataHub's graph via the MCP server / Skills layer, a scorer that computes per-asset signatures and anomaly scores across the nine dimensions, a narrative writer that turns a flagged walk plus DataHub ownership/docs context into an actionable alert, and an alert writer that persists the narrative and stamps ogle-drift-flagged (plus per-severity ogle-drift-high) tags back onto the graph.
Between runs, Ogle keeps a persistent baseline + incident memory (the BaselineStore): last known-good signature per asset, fingerprints of incidents already paged on, operator-muted assets — so a scheduled loop pages once per incident instead of every tick. That store sits behind one seam built to swap onto the salience-ranked, so past false positives and real incidents sharpen future walks.
The store is corruption-resilient — a bad JSON blob from a truncated cloud-sync or foreign version is quarantined to .corrupt.N, a loud warning goes to stderr, and the run re-baselines rather than crash-looping silently blind to drift.
The whole suite is keyless and Docker-free to test — every network call is monkeypatched — so pytest -q runs green from a fresh clone with no DataHub and no API key.
Challenges we ran into
- Scoping drift to what matters. A naïve diff flags every table that moves. Ogle scores against the serving path and severity so a Monday-bouncing dashboard doesn't page while a genuine schema + volume + quality shift on a serving table does — verified with the offline demo, which drifts one table's serving-path signals while leaving a sibling clean to prove the scoping.
- Silent-blind failure modes. The worst thing a monitor can do is go quiet and look healthy. That drove three defenses: the corruption quarantine above, a
--fail-on-unreachablegate so a total DataHub outage exits non-zero instead of reading as "all clear," and anogle_store_age_secondsheartbeat gauge so Prometheus alerts fire when Ogle itself stops running. - Making the judge path actually runnable. Early quickstarts referenced scripts that didn't exist; rewrote to verified commands and shipped a keyless, reproducible offline drift demo (
ogle demo) captured verbatim inexamples/alerts/. - DataHub Quickstart on Windows/WSL + write-back maturity — documented workarounds in
docs/live-verification.mdanddocs/DEPLOY.md.
Accomplishments that we're proud of
- 1,500+ passing tests, keyless.
pytest -qruns green from a fresh clone — no DataHub, no API keys, no Docker. The judge's first five minutes never break. - Nine drift dimensions on one signature. Schema, volume, quality, distribution, mean, stdev, range, shape, freshness — each a distinct failure mode with its own labeled alert. Most drift detectors ship one or two.
- One keyless command reproduces the flagship alert.
ogle demoseeds baselines, fires a HIGH serving-path incident spanning 7 of 9 dimensions in one alert, and reproducesexamples/alerts/churn-orders-drift.mdverbatim. Zero setup. - Memory that makes the second week better than the first. Mute-with-reason, snooze-with-expiry, incident fingerprints, severity × serving × ownership × recurrence cross-tabs — designed so the on-call inherits every triage judgment the last on-call made.
-** First-class observability.** Prometheus text-exposition endpoint (
ogle metrics) with the apex cross-tabogle_incidents_serving_unowned_recurring— the "page-before-everything" signal a flat count structurally can't express.
What we learned
Statelessness is the ceiling on most lineage tooling. The moment an agent remembers which alerts were noise, its signal-to-noise flips from "another dashboard" to "a triage assistant." And a hackathon judge's first five minutes are the demo: a keyless, no-Docker reproduction of the core alert is worth more than any architecture diagram — so we invested in making pytest and ogle demo run clean from a fresh clone before we invested in anything else.
What's next for Ogle
Memory and reach. Swapping the BaselineStore onto my Aegis MemoryAgent seam so salience-ranked forgetting sharpens future walks; agent-to-agent Ogle deployments that share incident memory across teams so one team's confirmed drift primes another's triage; and publishing Ogle's DataHub Skill wrapper back upstream as an OSS contribution.
Log in or sign up for Devpost to join the conversation.