GraphMedic

A self-healing loop for DataHub: when an AI agent fails, GraphMedic diagnoses the metadata gap, drafts a fix, and proves it worked

Comment

Inspiration

AI agents that read a data catalog are only as good as the catalog itself. A text-to-SQL agent doesn't fail because its code is wrong; it fails because a column has no description, no glossary term, no owner. Today, someone notices the wrong answer, opens a ticket, and a human manually fixes the metadata days later. We wanted to close that loop automatically: catch the failure, diagnose exactly what's missing, draft the fix, and prove it actually worked, without breaking the "no code deploys itself" rule responsible teams have for touching production metadata.

What it does

GraphMedic watches a real text-to-SQL agent answer questions against a DuckDB warehouse, using DataHub (via the official MCP Server) as its only source of context: table descriptions, column descriptions, glossary terms, ownership.

When the agent fails (bad SQL, an empty result, or the model itself reporting low confidence), a deterministic Referee (no LLM, just rules) diagnoses exactly which piece of metadata is missing or wrong. A second agent drafts a specific, evidence-cited patch via Gemini, with a confidence score. A human reviews the patch, current context next to the proposed fix, full reasoning, and approves or rejects it. On approval, GraphMedic writes the fix back to DataHub through the MCP Server's mutation tools, then automatically re-runs the exact question that failed. If it now passes, that's recorded as proof, not just a claim.

Every rejection is remembered too: the next patch attempt for the same gap type gets shown recent rejection reasons, so a human's correction actually changes future behavior.

Signed-in users get their own workspace: their own dataset (a local DuckDB file, or a live postgresql:// / mysql:// database DuckDB attaches to), their own Gemini key, their own DataHub identifiers, not just one shared demo.

How we built it

FastAPI + server-rendered Jinja2 templates, DuckDB as the query engine, Google's Gemini for both the SQL-writing agent and the patch-drafting agent, and the official mcp-server-datahub MCP Server for every DataHub read and write. GraphMedic never talks to DataHub's API directly. Per-user configuration (dataset, credentials) is threaded through a single Python contextvars.ContextVar, since FastAPI runs request handlers in a threadpool and a plain global would leak between concurrent users. Auth is email + password with PBKDF2 hashing, no OAuth, a deliberate scope cut given the timeline. 144 tests cover the diagnoser's rule logic, the write-back/re-run loop, and the web app's routes.

Challenges we ran into

Making "point GraphMedic at a different dataset" actually true, not just a claim: the DataHub platform/db fields only control which URN gets looked up, completely separate from where SQL actually executes, which was originally a fixed local DuckDB connection. We added real support for postgresql:// and mysql:// connection URLs (DuckDB's ATTACH) so a signed-in user's own database genuinely works, not just the demo warehouse.

Keeping per-user config safe under concurrent requests without a database, solved with a single contextvar rather than mutable global state.

Deciding what NOT to build given the timeline: no OAuth, no per-user DataHub server (the MCP Server connects to one DataHub instance at process start), and saying so plainly in the product's own UI copy, not just in this writeup.

What we're proud of

The fix isn't just applied, it's proven. GraphMedic re-runs the original failing question after every approval and shows PASSED or STILL_FAILING, which is a stronger claim than "we wrote a tag back."

Built With

Share this project:

Updates