DataMesh Warden

Inspiration

In modern data architectures, data pipelines break silently: an upstream microservice changes a payload schema, a critical field receives unexpected NULLs, or an ingestion job crashes. Data engineers often spend hours manually parsing messy stack traces, isolating corrupted rows, writing corrective SQL migrations, and verifying test runs in sandbox datasets.

We built DataMesh Warden to transform data operations from manual, reactive firefighting into an autonomous, self-healing workflow powered by Google Cloud and Gemini 3.5

What it does

DataMesh Warden is an event-driven, autonomous AI platform that monitors BigQuery and SQL pipelines, diagnoses data anomalies, and coordinates safe remediations in real time.

When an incident or schema violation occurs, the system's Agent Harness coordinates three specialized sub-agents:

• Log & Schema Investigator Agent: Ingests raw error logs, analyzes schema mismatches, and isolates root causes (e.g., nullability constraint failures, datatype drift).
• Patch & Sandbox Validation Agent: Generates corrective SQL DDL/DML scripts and tests them against an isolated BigQuery sandbox dataset to guarantee zero regressions.
• Governance & Policy Gatekeeper Agent: Validates proposed patches against enterprise compliance rules (e.g., blocking destructive DROP TABLE operations and verifying least-privilege IAM policies).

All agent thoughts, tool execution steps, and visual schema diffs stream live to an interactive Streamlit "Incident War Room" dashboard, enabling one-click human-in-the-loop review and execution.

How we built it

• AI Orchestration & Reasoning: Developed in Python 3.11 using the Google Gen AI SDK (google-genai) with Gemini 3 Pro serving as the master orchestrator.
• Multi-Model Pipeline: Integrated Gemma on Cloud Run to handle high-throughput edge log tokenization and anomaly filtering alongside Gemini 3 Pro.
• Agent Memory & State Management: Google Cloud Firestore tracks live execution traces, tool calling states, and incident audit histories.
• Data Warehouse Integration: Direct integration with Google Cloud BigQuery for automated schema inspection, table cloning, and sandbox dry-run testing.
• Harness & UI Deployment: Built with a FastAPI agent harness running on Cloud Run and a dark-themed Streamlit dashboard.

Challenges we ran into

• Ensuring Data Safety in Autonomy: AI models cannot be given unrestricted access to production databases. We solved this by creating a sandboxed execution harness where patches are strictly validated in isolated dataset clones before human approval.
• Asynchronous Multi-Agent State Synchronization: Coordinating multi-turn sub-agent tool calls across Cloud Run instances required designing a resilient state-persistence layer backed by Firestore.

Accomplishments that we're proud of

• Reduced typical data pipeline incident Mean Time to Resolution (MTTR) from ~45 minutes to under 30 seconds.
• Built a self-healing pipeline where AI-generated SQL is rigorously tested against schema sandboxes before presentation.
• Delivered an end-to-end multi-agent system fully deployed on Google Cloud infrastructure.

What we learned

• Best practices for multi-agent tool routing using the unified google-genai Python SDK.
• How to balance deterministic data engineering safeguards with autonomous LLM reasoning.

What's next for DataMesh Warden

• Native connectors for Apache Airflow, Cloud Composer, and dbt Cloud.
• Automated downstream impact analysis for BI tools like Looker.
• Proactive data drift detection using streaming Pub/Sub ingestion.

Built With

• aiops
• data-engineering
• docker
• fastapi
• gemini-3
• gemma
• google-cloud
• google-cloud-bigquery
• google-cloud-firestore
• google-cloud-run
• google-genai
• pydantic
• python
• sql
• streamlit