Data Pilot

DataPilot is an AI-powered data engineering platform that autonomously investigates pipeline failures, traces data lineage, and generates production-ready code fixes

Comment

Inspiration

Data engineers and data stewards waste countless hours manually tracing lineage and debugging broken pipelines. When a critical business dashboard shows stale data, finding the root cause across a modern data stack (dbt, Snowflake, Postgres, Looker) is like finding a needle in a haystack. We were inspired to build an autonomous AI agent that leverages DataHub's metadata to instantly identify root causes, trace lineage, and even suggest code fixes, freeing up engineers to focus on building rather than debugging.

What it does

DataPilot is an autonomous AI Data Engineer that integrates seamlessly with DataHub's Model Context Protocol (MCP) server.

  • AI Investigation: Users can ask plain English questions like "Why is the Sales Dashboard broken?" DataPilot analyzes the live MCP context (schema changes, failing freshness assertions, and lineage graphs) to pinpoint the exact failure (e.g., a missing column in a dbt model).
  • Enterprise Governance (Organizations): A robust workspace manager allowing teams to map workspaces to Data Domains (e.g., E-Commerce Data Mesh), manage granular DataHub IAM policies, and view detailed Audit Logs.
  • Crowdsourced Data Quality (Communities): A central hub where data practitioners can upvote discussions, publish "Gold Certified" data assets for peer review, and request missing datasets directly from data stewards.

How we built it

  • Frontend Framework: Next.js 16 (Turbopack) with React and TypeScript for a blazing-fast, server-rendered application.
  • Styling: Tailwind CSS 4 to craft a "Full Strength," enterprise-grade, glassmorphic UI without heavy component libraries.
  • AI Engine: Google Gemini 2.5 Flash (via the new @google/genai SDK) to provide ultra-low latency, high-reasoning responses.
  • Metadata Layer: We structured DataHub metadata (lineage, data quality assertions, governance tags) into an XML-like structure via the Model Context Protocol (MCP), grounding the AI and preventing hallucinations.
  • Deployment: Deployed globally on Vercel for instant access.

Challenges we ran into

  • AI Hallucinations: Initially, the AI would guess why a pipeline failed. We solved this by rigorously engineering the System Prompt using MCP to inject a strict, structured graph of <ASSET>, <LINEAGE_TRACE>, and <SCHEMA_CHANGE> data.
  • SDK Integration: Migrating to the latest Google Gen AI SDK in a Next.js App Router environment required careful environment variable mapping and custom Route Handler configurations.
  • Designing for Enterprise: Hackathon projects often look like simple tools. We challenged ourselves to build an interface that sponsors and judges would recognize as a legitimate Enterprise SaaS product—requiring us to build complex state management for RBAC panels, audit logs, and interactive communities from scratch.

Accomplishments that we're proud of

  • Lightning-Fast Reasoning: By pairing Next.js App Router with Gemini 2.5 Flash, the agent traces complex data lineage graphs and generates answers in seconds.
  • Zero-Hallucination Architecture: Our prompt engineering securely restricts the agent to the provided DataHub MCP context.
  • The "DataPilot" UX: We successfully built an investigation interface where users aren't just chatting with a bot; they are applying Environment (PROD/DEV) and Platform (Snowflake, dbt) filters that dynamically update the AI's investigation parameters.

What we learned

  • The Model Context Protocol (MCP) is the missing link for enterprise AI. By feeding the LLM a structured subgraph of DataHub metadata, its usefulness skyrocketed.
  • Gemini 2.5 Flash is an absolute powerhouse for autonomous agent workloads where speed and complex chain-of-thought reasoning must happen simultaneously.
  • Building a UI that "strengthens everyone on the platform" (data engineers, analysts, and stewards) requires treating Data Governance not as a chore, but as a collaborative, community-driven effort.

What's next for DataPilot

  • Two-Way MCP Integration: Moving beyond just reading metadata to executing mutations (e.g., automatically tagging a dataset as PII or resolving a Data Quality incident directly from the chat).
  • Automated Pull Requests: Upgrading the "Generate PR" action button to securely connect to GitHub and autonomously open a fix for a broken dbt model.
  • Slack/Teams Integration: Bringing the DataPilot investigation agent directly into the communication platforms where data alerts actually happen.

Built With

Share this project:

Updates