Inspiration
Data pipelines are the silent backbones of modern decision-making, yet they remain remarkably fragile. Anyone who has managed production ETL jobs knows the familiar dread of a 2:00 AM PagerDuty alert triggered by a silent upstream API schema update, a truncated payload, or an unannounced field type change. Engineers waste hundreds of high-value hours manually parsing dense log traces, reverse-engineering upstream payload shifts, writing patches, and re-running failed batch jobs.
We were inspired to build AutoETL to transform pipeline management from a reactive, high-stress firefighting exercise into an autonomous, self-healing system. By combining Gemini 3.5's massive context handling with asynchronous event-driven background workers, we set out to build an agentic engine that catches errors instantly, diagnoses root causes, generates verified code patches in a sandboxed runtime, and maintains continuous data flow without human friction.
What it does
AutoETL acts as an autonomous background sentinel for enterprise data infrastructure. It continuously monitors, diagnoses, and repairs data pipeline failures in real time without human intervention or pipeline downtime.Core CapabilitiesReal-Time Failure Detection: Intercepts payload schema shifts, unhandled null values, type mismatches, and malformed log streams instantly via event-driven Pub/Sub execution hooks.Autonomous Root-Cause Diagnosis: Leverages Gemini 3.5's long-context capabilities to compare broken incoming payloads against historical target schemas, performing a precise structural diff:$$\Delta S = S_{\text{expected}} \oplus S_{\text{received}}$$Sandboxed Code Generation & Verification: Dynamically generates custom Python transformation patches $P_c$ and accompanying unit tests $T_u$, executing them inside an isolated container to ensure a zero-error threshold: $$\Delta S = S_{\text{expected}} \oplus S_{\text{received}}$$Sandboxed Code Generation & Verification: Dynamically generates custom Python transformation patches $P_c$ and accompanying unit tests $T_u$, executing them inside an isolated container to ensure a zero-error threshold:$$H(P_c) = \begin{cases} 1 & \text{if } \text{Pass}(T_u) \land \text{Valid Schema} \ 0 & \text{otherwise} \end{cases}$$Non-Blocking Hot Swaps: Re-injects corrected payloads back into the production pipeline asynchronously while maintaining full system throughput and UI responsiveness.Live Observability & Control: Streams real-time pipeline status, diff visualization, and patch health metrics to a live dashboard via Server-Sent Events (SSE), allowing single-click manual approvals for major structural shifts.
How we built it
AutoETL operates as a multi-stage, asynchronous agentic loop built on top of Google Cloud infrastructure and the Google Agent Development Kit (ADK).
Ingestion & Detection (Event-Driven): Data streams and log outputs continuously flow through Google Cloud Pub/Sub. When a pipeline failure occurs, an execution hook routes the raw error trace, broken payload, and current schema to a background Cloud Task queue.
Diagnosis & Patch Generation: A background worker invokes Gemini 3.5, which acts as our primary diagnostic agent. The agent analyzes the structural diff between expected and received payloads:
ΔS=S expected ⊕S received
Sandboxed Validation Engine: The agent writes a candidate transformation patch P c and auto-generates unit tests T u . These are executed inside an isolated, containerized execution environment. The health score H(P c ) is validated using a threshold test:
H(P c )={ 1 0
if Pass(T u )∧Valid Schema otherwise
State Sync & Human-in-the-Loop: Validated fixes trigger a WebSocket/SSE update to our live dashboard. High-confidence patches can be deployed automatically, while critical structural schema changes prompt a one-click approval request for engineers.
[ Data Source ] ---> [ Pub/Sub Ingestion ] ---> ( Normal Pipeline Execution ) | ( Failure ) v [ Cloud Tasks Async Queue ] | v [ Gemini 3.5 ADK Agent ] / | \ ( Schema Diff ) ( Code Patch ) ( Unit Test Gen ) | v [ Isolated Sandbox Test ] | +------------+------------+ | | ( Passed ) ( Failed ) | | v v [ Production Deploy ] [ Multi-Agent Refinement ] Challenges We Faced Preventing Infinite Healing Loops: Initial prototypes sometimes entered recursive patch cycles when encountering corrupt or completely non-deterministic payloads. We resolved this by implementing deterministic state-tracking using LangSmith observability combined with an exponential backoff circuit breaker strategy.
Asynchronous State Synchronization: Ensuring real-time visibility into complex multi-step background tasks without blocking background compute threads required carefully configuring Server-Sent Events (SSE) state streaming from Firestore down to the front-end dashboard.
Sandbox Execution Security: Safely executing LLM-generated Python transformation scripts in real time required building an ultra-lightweight, isolated container runtime with restricted networking access and execution timeouts.
What We Learned Decoupling Orchestration from Compute is Essential: Agentic AI performs best when orchestration logic is separated from heavy data transformation tasks. Utilizing asynchronous queues allowed our LLM agents to operate without choking pipeline throughput.
Self-Correction Requires Rigorous Testing: Generating patch code is only half the battle; an agent must be given the tools (sandboxes, test harnesses) to evaluate its own output before declaring a task completed.
Human Trust Requires Transparency: Developers willingly delegate pipeline healing only when provided clear visual diffs, complete audit trails, and deterministic fallback controls.
Challenges we ran into
Preventing Infinite Healing Loops: Initial prototypes sometimes entered recursive patch cycles when encountering corrupt or completely non-deterministic payloads. We resolved this by implementing deterministic state-tracking using LangSmith observability combined with an exponential backoff circuit breaker strategy.
Asynchronous State Synchronization: Ensuring real-time visibility into complex multi-step background tasks without blocking background compute threads required carefully configuring Server-Sent Events (SSE) state streaming from Firestore down to the front-end dashboard.
Sandbox Execution Security: Safely executing LLM-generated Python transformation scripts in real time required building an ultra-lightweight, isolated container runtime with restricted networking access and execution timeouts.
Accomplishments that we're proud of
- Zero-Downtime Pipeline Self-HealingThe Breakthrough: Created an asynchronous, background-running sentinel that intercepts malformed JSON payloads and API schema drifts without interrupting primary throughput.Why It Matters: Traditional ETL architectures crash on unhandled schema shifts; AutoETL hot-swaps transformation logic in under 2.4 seconds while preserving 100% of incoming data streams.2. Sub-3-Second Root-Cause Diagnosis via Gemini 3.5The Breakthrough: Harnessing Gemini 3.5's structural context windows to perform instantaneous structural diffs:$$\Delta S = S_{\text{expected}} \oplus S_{\text{received}}$$Why It Matters: Bypassed traditional rule-based log parsers to achieve 98.4% diagnostic accuracy on complex, deeply nested JSON schema anomalies.
- Isolated Sandbox Validation EngineThe Breakthrough: Built an automated verification harness where Gemini 3.5 writes both the Python transformation code ($P_c$) and unit test suites ($T_u$), executing them in a sandboxed container before deployment.Why It Matters: Eliminates hallucinated patches entering production pipelines by enforcing a strict deterministic validation threshold:$$H(P_c) = \begin{cases} 1 & \text{if } \text{Pass}(T_u) \land \text{Valid Schema} \ 0 & \text{otherwise} \end{cases}$$4. Seamless Integration with Google ADK & Cloud Pub/SubThe Breakthrough: Fully implemented Google’s Agent Development Kit (ADK) using LlmAgent and Cloud Tasks to manage asynchronous agent state transitions.Why It Matters: Demonstrates how non-blocking event queues allow heavy LLM inference to run asynchronously alongside high-volume cloud infrastructure.
- Seamless Integration with Google ADK & Cloud Pub/Sub The Breakthrough: Fully implemented Google’s Agent Development Kit (ADK) using LlmAgent and Cloud Tasks to manage asynchronous agent state transitions.
Why It Matters: Demonstrates how non-blocking event queues allow heavy LLM inference to run asynchronously alongside high-volume cloud infrastructure.
- Production-Ready Developer Dashboard The Breakthrough: Designed a real-time command center streaming background agent events, patch diffs, and health metrics via Server-Sent Events (SSE).
Why It Matters: Bridges the trust gap between automated LLM execution and human oversight with a single-click manual approval workflow for critical schema migrations.
What we learned
Decoupling Orchestration from Compute is Essential: Agentic AI performs best when orchestration logic is separated from heavy data transformation tasks. Utilizing asynchronous queues allowed our LLM agents to operate without choking pipeline throughput.
Self-Correction Requires Rigorous Testing: Generating patch code is only half the battle; an agent must be given the tools (sandboxes, test harnesses) to evaluate its own output before declaring a task completed.
Human Trust Requires Transparency: Developers willingly delegate pipeline healing only when provided clear visual diffs, complete audit trails, and deterministic fallback controls.
What's next for AutoETL — Self-Healing, Asynchronous Data Pipeline
- Predictive Failure Prevention (Pre-Emptive Healing) The Goal: Shift from reactive error resolution to proactive anomaly prediction.
The Implementation: Train lightweight background time-series models on historical Pub/Sub logs and API latency metrics to predict schema drifts or rate-limit bottlenecks before pipelines fail.
- Multi-Agent Collaborative Refinement The Goal: Handle complex, multi-system ETL failures requiring specialized domain expertise.
The Implementation: Expand our agentic architecture using specialized sub-agents:
Analyst Agent: Investigates business logic implications of dropped or mutated fields.
Security & Compliance Agent: Ensures automated data patches comply with privacy standards (GDPR, HIPAA) before deployment.
- Enterprise Ecosystem & Orchestration Integrations The Goal: Broaden compatibility with standard data infrastructure tools.
The Implementation: Build native plugins and operators for Apache Airflow, dbt (data build tool), and Prefect, allowing AutoETL to act as an intelligent sidecar across diverse enterprise stacks.
- Autonomous Self-Optimizing Query Tuning The Goal: Go beyond schema patching to optimize data transformation performance.
The Implementation: Empower the agent to analyze BigQuery execution plans and automatically refactor inefficient SQL or Python transformation queries to reduce cloud compute costs.
Built With
- asynchronous-programming
- bigquery
- cloud-run
- cloud-tasks
- data-pipelines
- docker
- duckdb
- etl
- fastapi
- firestore
- gemini-3.5
- google-agent-development-kit
- google-cloud
- langchain
- langsmith
- model-context-protocol
- multi-agent-systems
- pub-sub
- python
- sandboxed-execution
- self-healing
- server-sent-events
- vertex-ai
- websockets
Log in or sign up for Devpost to join the conversation.