DataOps SWAT Team

When your data pipeline breaks at 2am, the SWAT team is already on it.

Comment

Inspiration

Data engineers spend 4+ hours manually tracing a broken dashboard back to a renamed column in an upstream dbt model. The lineage exists in DataHub but nobody has written the agent to read it automatically. We built the DataOps SWAT team to close that gap.

What it does

The SWAT Team is a multi-agent system that autonomously detects, diagnoses, engineers fixes for, and validates data pipeline incidents—all using DataHub as the single source of metadata truth. 4 specialized agents coordinate through a shared state machine:

  1. Sentry Agent—Scans DataHub metadata and detects anomalies (schema drift, freshness violations, ownership gaps, lineage orphans)
  2. Detective Agent—Traces lineage upstream to identify root causes with confidence scores
  3. Engineer Agent—Generates SQL fixes using real DataHub schema context + LLM (OpenRouter/Ollama)
  4. Validator Agent — Validates fixes against downstream lineage before deployment

A Streamlit Mission Control dashboard shows real-time pipeline progress, incident details, lineage graphs, and fix previews.

How we built it

  • Async Python with Pydantic v2 for type safety across all agents
  • DataHub GraphQL API via a custom async MCP client that queries datasets, schemas, lineage, and ownership
  • State machine for incident lifecycle management (DETECTED → DIAGNOSING → ROOT_CAUSE_IDENTIFIED → FIXING → FIX_PROPOSED → VALIDATING → READY_TO_DEPLOY)
  • LLM gateway (OpenRouter/Ollama) for metadata-aware SQL code generation
  • Streamlit for the mission control dashboard

Challenges we ran into

Infrastructure was the hardest part. DataHub's full stack needs 8GB+ RAM. We tried GitHub Codespaces (32GB disk limit), local Docker (8GB laptop), Oracle Cloud (no free capacity), and Azure (size restrictions) before landing on a DigitalOcean droplet. The lesson: persistence matters more than perfect infrastructure.

Accomplishments that we're proud of

  • Real SQL generated from real DataHub schemas—not mock data, not hallucinated columns. The Engineer Agent reads actual schema fields before writing ALTER TABLE statements.
  • Lineage-aware validation that catches breaking changes before deployment. The Validator Agent checks if a dropped column is used downstream.
  • 93 tests covering all 4 agents, the orchestrator, and the MCP client.
  • Full async pipeline from detection to validated fix in under 5 minutes.

What we learned

  • DataHub's GraphQL schema is powerful but requires live introspection against a running instance
  • Multi-agent orchestration needs strict state machines to prevent chaos
  • Asyncio + Pydantic is a sweet spot for agent systems in Python

What's next for DataOps SWAT Team

  • Slack/Teams notifications for new incidents
  • Auto-create GitHub PRs with generated SQL fixes
  • dbt model generation from DataHub schemas
  • Integration with Great Expectations for data quality rules

Built With

  • asyncio
  • datahub
  • httpx
  • ollama
  • openrouter
  • pydantic
  • python
  • sqlparse
  • streamlit
Share this project:

Updates