Recall

A camera feeds bad training data for three weeks, which robots learned from it? Recall walks DataHub lineage forward, stops exactly those machines and writes the recall into the catalog.

Comment

Inspiration

Most ML "safety" tools walk backward from a broken model. The question warehouses and fraud teams actually ask is the opposite: a data source went bad for three weeks — which deployed things learned from it, and which ones are still clean?

That answer lives across a gap we measured on DataHub GMS v1.6.0. get_lineage through the Agent Context Kit returns dozens of hops at the dataset layer… and zero hops from training set → model → deployment. The edges are stored in aspects. Agents just can't traverse them. An empty lineage result looks like "nothing's affected." That's a lie during an incident.

Recall exists to close that gap — and to make the answer operational, not a PDF.

What it does

  1. Notices when a source starts producing bad data (cheap thresholds — no LLM every tick).
  2. Investigates via DataHub's Agent Context Kit; every date boundary needs cited evidence.
  3. Decides with a deterministic kernel — no LLM in the blast-radius math. Example: 47 robots grounded, 71 spared; naive downstream stops 118.
  4. Gates autonomy with policy (max machines, max site share). Too big → escalate and wait.
  5. Writes back into DataHub: recalled tag, banner on the model, deployments flipped to OUT_OF_SERVICE, recall Document via ACK.
  6. Interlock — the unit reads catalog status URN-direct and refuses to arm.
  7. Recovers when the source is healthy and a clean model has shipped.

Same engine, second domain: labelling vendor → fraud scorer → API endpoints → 31 grounded / 54 naive / 23 spared, zero kernel changes.

Invariant: The agent proposes. A deterministic kernel decides. No model ever decides whether a machine is allowed to move.

How we built it

  • DataHub Agent Context Kit for agent reads + save_document / add_tags
  • Native aspects for MLModelDeployment status (IN_SERVICE / OUT_OF_SERVICE)
  • Pure deterministic kernel + policy + Sentinel loop
  • Recall MCP server for the hops DataHub can't traverse
  • DataHub Skill (skills/datahub-recall/) teaching the workflow and the lineage gap
  • Offline-first path: ./run.sh, hosted demo, 39 tests — judges don't need Docker for the core claim

Challenges we ran into

  • ACK / get_lineage silently returns empty across the ML hop — easy to misread as all-clear
  • MLModelDeployment isn't a GraphQL entity; UI pages 404 even when aspects are correct
  • Search index lags after writes — all status reads are URN-direct
  • Keeping the LLM out of the decide path without turning the project into a report generator

Accomplishments that we're proud of

  • End-to-end live loop: investigate → write-back → interlock GROUNDED (see examples/sentinel-live-datahub.txt)
  • Measured lineage-gap finding with repro: docs/upstream-finding.md
  • Upstream issue: datahub#19061
  • Skills PR: datahub-skills#139
  • Domain-general engine (robots and fraud endpoints), 39 tests, public Apache-2.0 demo

What we learned

Metadata only helps agents if they can traverse it. DataHub already stores model→deployment; projecting it into GraphQL and lineage would unlock a whole class of production safety agents. Until then: aspect-direct reads + write-back as a control plane is the workable pattern.

What's next for Recall

Land the skills PR, push the GraphQL / DeployedTo projection fix, and wire continuous perception against live assertion / run-event streams — not just synthetic fleet quality.

Built With

  • agent
  • anthropic
  • claude
  • context
  • datahub
  • docker
  • fastapi
  • kit
  • mcp
  • pytest
  • python
  • pyyaml
  • render
Share this project:

Updates