DataHub Sentinel — Autonomous Anomaly Detection Agent

The Problem

Data pipelines fail silently every day. A schema changes upstream, data quality degrades, or latency spikes—but downstream teams don't know until users complain. Existing monitoring tools show what broke, but not why or who to notify.

The Solution

DataHub Sentinel is an autonomous agent that:

1. Detects anomalies in real-time metrics (latency, error rates, schema drift)
2. Queries DataHub via MCP Server for lineage context (what's upstream? what's downstream?)
3. Identifies root cause using metadata (recent changes, ownership, schema history)
4. Generates incident reports with actionable recommendations
5. Writes back to DataHub to enrich the knowledge graph

Unique Angle

Unlike existing DataHub MCP servers that passively answer queries, Sentinel is active—it monitors, detects, investigates, and reports. It combines anomaly detection with metadata context to provide not just alerts, but actionable intelligence.

How It Works

1. Sentinel monitors data pipeline metrics via a lightweight collector
2. When an anomaly is detected, it queries DataHub's MCP Server for lineage context
3. It correlates the anomaly with recent schema changes, ownership info, and downstream dependencies
4. It generates a structured incident report with root cause analysis and recommended actions
5. The report is written back to DataHub as an incident annotation

Tech Stack

• DataHub MCP Server — queries lineage, schema, ownership metadata
• Python/FastAPI — backend API server
• Statistical anomaly detection — Z-score + IQR methods
• Vercel — free-tier deployment
• Apache 2.0 license — fully open source

API Endpoints

Try It Live

Visit the demo URL and use the interactive dashboard to:

• Query dataset lineage through the MCP interface
• Simulate anomaly detection with custom metrics
• Generate incident reports with root cause analysis
• View trust scores for datasets

What's Real vs Mocked

• ✅ Real: MCP server protocol, lineage queries, anomaly detection algorithms, incident report generation, trust score calculation
• 🔄 Simulated: Metric data feeds (simulated pipeline metrics for demo), DataHub backend (mocked responses matching real DataHub API schema)

Challenge Track

Track 1: Agents That Do Real Work — Sentinel finds the problem, investigates root cause via DataHub, and writes results back before anyone downstream feels it.

Built With

• ai-agents
• anomaly-detection
• datahub
• fastapi
• mcp-server
• python
• vercel