ContextPilot AI

AI Chief Data Officer powered by DataHub. Multi-agent AI that investigates incidents, analyzes lineage, simulates data outages, and improves governance to help data teams prevent failures.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

Inspiration

Modern AI agents often fail because they lack context about enterprise data. Data engineers spend hours tracing lineage, investigating incidents, checking governance issues, and estimating the impact of data failures.

We built ContextPilot AI to act as an AI Chief Data Officer that understands the entire DataHub metadata graph and helps teams make faster, safer decisions.

What it does

ContextPilot AI is a multi-agent platform powered by DataHub.

It includes specialized AI agents for:

  • Incident Investigation
  • Lineage Analysis
  • Crisis Simulation
  • Metadata Analysis
  • Governance Recommendations
  • Executive Reporting
  • AI Code Generation

The flagship feature is Crisis Simulation, which predicts the downstream business impact before a dataset fails.

Instead of reacting after production incidents happen, teams can proactively understand risk and take action.

How we built it

Our stack includes:

  • Next.js 14
  • TypeScript
  • Tailwind CSS
  • FastAPI
  • Python
  • WebSockets
  • DataHub Metadata Graph
  • GraphQL
  • Multi-Agent AI Architecture

The application works entirely in Mock Mode for demonstration while keeping the same architecture required for a real DataHub deployment.

Challenges we ran into

Designing a realistic metadata graph that could demonstrate lineage, governance, ownership, schema evolution, and downstream impact required significant planning.

We also designed multiple AI agents that communicate through a central Chief Data Officer agent while keeping the architecture modular and extensible.

Accomplishments

  • Built a complete multi-agent AI platform
  • Created seven production-quality application screens
  • Implemented lineage-aware incident investigation
  • Added AI-powered governance recommendations
  • Developed a unique Crisis Simulation engine
  • Generated executive governance reports
  • Open-sourced the entire project under Apache 2.0

What we learned

We learned how valuable DataHub becomes when used as the knowledge layer for AI agents. Rich metadata dramatically improves reasoning quality compared to traditional LLM workflows.

What's next

Future work includes:

  • Live DataHub MCP integration
  • Real-time governance write-back
  • Slack and Microsoft Teams integration
  • Autonomous remediation agents
  • Production deployment with enterprise authentication ## What's next for ContextPilot AI

Built With

Share this project:

Updates