Preflight

What it does

This is a submission to the Agents That Do Real Work challenge.

Preflight audits an open-source DataHub catalog for whether an AI agent could actually use it, scores every table 0 to 100 against a deterministic rubric, and queues the fixes behind human approval. The agent never applies a change on its own.

The part that matters is what happens when the agent hits something it cannot infer. It does not guess. It asks one plain question with a short list of options and an honest out ("Unsure, not my call"). The answer folds into an operating doc as a convention, and the question never comes back.

Answer one question about one column, and Preflight stops asking that question everywhere. A human says once what release_speed means on raw_pitches. On the next table the agent asks nothing about release_speed or about effective_speed, a sibling column on a table nobody answered for. The convention transferred on a shared term, not a memorized field.

The silence is tested, not narrated. tests/test_demo_pair.py builds a counterfactual world where the first column already had a description, so the agent never had a reason to ask there. In that world the card jumps to the second table instead. Add the same convention and that world goes quiet too. A test that cannot fail proves nothing, so this one was built to fail first.

How I built it

Python agent plus a Next.js review UI, on a reusable spine:

• Read through the MCP Server from DataHub's Agent Context Kit (mcp-server-datahub, the Kit's tool surface): search, get_entity, get_lineage.
• Score with a deterministic rubric (description 30, lineage 25, freshness 20, ownership 15, glossary 10). A missing table description caps the score at 39 no matter what else is present, because an agent cannot substitute inference for the one fact only a human has.
• Propose every change into an approval queue. The agent has no write path around it.
• Ask and learn through elicitation cards and a fold protocol that writes conventions into an operating doc stored as a DataHub documentation aspect. The agent's memory lives in the catalog.
• Local only. Inference runs on Ollama qwen3.5:4b. The cloud fallback is a deliberate dead end that raises rather than calling a hosted provider, so no code path reaches one.

packages/spine/ is a reusable open-source DataHub write-back engine. It ships across two entries: this one scores a catalog with it, and datahub-spotter vendors the same engine read-only to catch silent ML pipeline failures. One engine, two products.

Challenges I ran into

DataHub v1.2.0.1's MCP get_entity carries no lastModified field, confirmed absent across all six live captures. Rather than invent a freshness score, the chip reads "no signal" and the rubric weight redistributes across the other four categories behind a named constant. The same discipline covers usage: no usage source exists in this world, so the ranking degrades to pure readiness behind a labeled uniform factor. Both are visible in the UI, not buried in code.

The live MCP search tool pages at 10 results. Our catalog grew past that, and the lowest-scoring table (ranked last, precisely because it had the least metadata) silently fell off page one. The client now paginates and fails loudly on any shortfall. A silent truncation in a tool whose whole job is finding under-documented tables would have hidden exactly the tables it exists to find.

Accomplishments I am proud of

Every number in the repo is committed output of a guarded script. tests/test_examples.py regenerates the entire examples/ folder in memory on each run and fails if a committed file has drifted from what the generator actually produces. The screenshots are real UI over real generated data, never mockups.

What I learned

A guard that cannot fail on the condition it exists to catch is not a guard. Before shipping any check I wrote down what a broken run produces, then confirmed the check went red on exactly that value. Several guards looked fine and asserted nothing.

What is next

After-state captures of approved writes landing in the live DataHub UI, and wiring the local model into description drafting, which is scoped and harnessed but not yet in the batch walk.

Open-source contribution to DataHub

Hit a defect in DataHub's own CLI during this build: datahub docker quickstart rejects any Docker Compose version 3 or later, because the version check is a literal string-prefix test for "2." that predates Compose's later major versions. A correct, current install is told it is running v1.

Another user had already opened an issue for the symptom. Rather than file a duplicate, I added the part that was missing from it: the root cause in the released code, confirmation on a second OS and Compose version, a pointer to the master-branch fix that has not yet shipped in a release, and a workaround verified against six version-string inputs including malformed and empty strings. https://github.com/datahub-project/datahub/issues/15763#issuecomment-5179837828

Full writeup: docs/upstream-compose-version-issue.md

Try it without installing anything

examples/ holds the entire loop as committed output, about 1,800 words end to end. Read five files in order and you have watched the agent score, ask, learn, and go quiet. No server, no Docker, no model.

Built With

• datahub
• mcp
• nextjs
• ollama
• playwright
• python
• typescript
• vercel