Data Doctor

Inspiration

Data quality incidents are rarely difficult because the problem itself is impossible to understand. They are difficult because the information needed to diagnose them is scattered across data catalogs, pipeline logs, schemas, query histories, dashboards, and ownership records.

When an alert such as "the dataset is stale" appears, a data engineer typically has to answer three questions: What went wrong? What is affected? And who needs to fix it?

We wanted to make that investigation process faster and more intelligent. That led us to build Data Doctor—an AI data engineer that uses DataHub as its source of truth and turns a natural-language incident into a structured investigation, diagnosis, blast-radius analysis, and remediation plan.

What it does

Data Doctor is an AI-powered Data Quality Incident Investigator. A user can describe an incident in plain English, and Data Doctor investigates the relevant data assets using live metadata from DataHub.

The system can:

• Identify the affected dataset and classify the incident as a freshness, quality, or schema-change problem.
• Retrieve dataset metadata, schemas, ownership information, assertions, tags, and lineage.
• Investigate upstream dependencies and downstream consumers to determine the potential blast radius.
• Use recent dataset queries and metadata to provide additional diagnostic context.
• Use an LLM to reason over the retrieved facts and generate a structured diagnosis.
• Clearly separate facts from inferences, making the AI's conclusions more transparent.
• Recommend targeted remediation actions, such as checking a specific pipeline or contacting the responsible team.
• Write investigation status back to DataHub using tags when mutation permissions are available.
• Fall back to a rules-based analysis engine when the local LLM is unavailable.

For the hackathon, we created two representative scenarios: a stale NYC Yellow Taxi dataset and a Healthcare Patient Records dataset containing ICD-10 formatting issues. These demonstrate how Data Doctor can identify root causes, assess downstream impact, and recommend next steps.

How we built it

Data Doctor was built as a full-stack application around the DataHub metadata ecosystem.

The frontend uses React/Next.js with Tailwind CSS to provide an incident input interface, investigation trace, diagnosis dashboard, recommendations, and lineage visualization.

At the center is an investigation orchestrator that coordinates the entire diagnosis workflow. Instead of creating custom API integrations for every DataHub operation, Data Doctor communicates with the official DataHub MCP Server. This gives the agent access to tools for searching assets, retrieving entities and schemas, exploring lineage, examining dataset queries, and writing tags back to the catalog.

The investigation pipeline follows four major stages:

1. Classify — Parse the natural-language incident and identify the target dataset and incident type.
2. Retrieve — Query DataHub through MCP for schemas, ownership, lineage, assertions, and other relevant metadata.
3. Reason — Provide the retrieved, real-world context to an LLM through a structured diagnosis prompt.
4. Compile — Validate the resulting JSON and transform it into facts, inferences, impact assessment, and recommended actions.

To keep the project accessible and hackathon-friendly, we designed it to run locally using DataHub, Docker, Ollama, and an open-weight model such as Llama 3 or Mistral, without requiring a paid AI API key. An offline demo mode also allows the interface and investigation experience to be explored without running the full backend stack.

Challenges we ran into

One of the biggest challenges was making an LLM useful without allowing it to invent metadata. Data quality investigations depend on accurate information about schemas, owners, dependencies, and pipelines, so hallucinated information could be worse than no information at all.

We addressed this by grounding the LLM exclusively in information retrieved from DataHub and explicitly separating factual observations from AI-generated inferences.

Another challenge was determining the blast radius of an incident. Data can flow through multiple datasets, dashboards, and pipelines, so investigating every possible dependency can quickly become expensive in terms of both execution time and LLM context. We therefore limited lineage traversal to three hops for the current implementation.

We also had to design the application to remain useful when parts of the AI stack are unavailable. Data Doctor therefore includes a rules-based fallback when Ollama or the selected model is unavailable, allowing the investigation experience to degrade gracefully instead of simply failing.

Finally, integrating AI reasoning with actual metadata mutations introduced another consideration: permissions. Writing investigation results back to DataHub depends on the configured GMS access token, so the application handles insufficient permissions gracefully and provides an alternative report workflow.

Accomplishments that we're proud of

We are particularly proud of turning a traditionally manual data-engineering workflow into an interactive AI investigation experience.

The project demonstrates that an LLM can do more than provide generic advice when it is grounded in a metadata platform. Data Doctor can combine catalog metadata, schema information, lineage, ownership, assertions, and query information to produce an investigation that is specific to the actual data environment.

We are also proud of the facts-versus-inferences design. Rather than presenting every AI conclusion as truth, Data Doctor makes the distinction explicit. This makes the investigation easier for engineers to validate and builds trust in AI-assisted troubleshooting.

The interactive investigation trace is another highlight. Users can see the system identify the incident type, search DataHub, retrieve the relevant metadata, reason over the evidence, and produce the final diagnosis instead of receiving an unexplained AI answer.

Most importantly, the project demonstrates a practical use of DataHub + MCP + local AI: using DataHub as the trusted context layer and AI as the reasoning layer on top of it.

What we learned

We learned that the value of an AI agent in data engineering comes less from the model itself and more from the quality and structure of the context provided to it.

DataHub provides a rich foundation for this because metadata, schemas, lineage, ownership, assertions, and other information can be brought together into a common context. MCP made that information accessible to the investigation agent through a standardized tool interface.

We also learned that trustworthy AI investigations require a clear distinction between what the system knows and what it believes. Grounding the model in retrieved metadata and explicitly separating facts from inferences makes the resulting system much more useful for engineering workflows.

Finally, we learned that good developer experiences matter just as much as the underlying AI. Features such as offline demo mode, investigation traces, graceful LLM fallback, structured results, and visual lineage make a technically complex system much easier to understand and use.

What's next for Data Doctor

The next step for Data Doctor is to move from investigation and recommendation toward automated remediation.

One major enhancement is column-level lineage blast-radius analysis. Instead of identifying only affected datasets and downstream assets, Data Doctor could determine exactly which columns and dashboard visualizations are likely to be impacted by a change or quality issue.

We also want to introduce automated remediation actions. For example, when Data Doctor identifies a failed Airflow pipeline, a user could potentially trigger a pipeline restart directly from the recommended remediation instead of manually switching to another system.

Beyond that, Data Doctor could evolve into a continuously running data reliability agent—monitoring incidents, investigating them automatically, notifying the right owners, tracking remediation, and recording the resolution back into DataHub.

Our long-term vision is simple: when a data incident happens, engineers should not have to search for the answer. Data Doctor should investigate it for them.

Built With

• antigravity
• datahub