Warden: Autonomous AI Incident-Resolution Agent

An autonomous AI on-call engineer that intercepts production alerts, traces root causes with Gemini, applies tested patches, and verifies system health — zero human intervention required.

Comment

💡 Inspiration

Every DevOps team faces the same nightmare: a production alert fires at 2 AM, someone gets paged, spends an hour investigating logs, finds the bug, writes a fix, tests it, deploys it, and verifies everything is back to normal — all while half-asleep. Most "AI on-call" tools stop at the diagnosis step. They summarize the alert and leave you to do the rest.

We wanted to build something different: an agent that acts like a true senior on-call engineer — one that doesn't just describe the problem, but takes action, verifies its own repairs, and learns from past incidents. Warden was born from that question: what would it look like if the entire incident response loop — from alert to resolution — required zero human intervention?

⚙️ What it does

Warden is an autonomous AI on-call agent that closes the full incident loop without waking anyone up:

  1. 🔔 Intercepts Alerts — Listens via HMAC-signed webhooks for CI test failures, error-rate spikes, unresponsive services, or flaky third-party integrations. Publishes each signal asynchronously to Google Cloud Pub/Sub for burst-safe, retry-guaranteed delivery.

  2. 🧠 Performs Deep Root-Cause Analysis — Queries a live AST-parsed Python call graph (stored in Firestore) to find functions implicated by the signal. Pulls precedent incidents from its memory bank. Feeds all of this into Gemini (via Google ADK) to generate a root-cause hypothesis and a blast-radius report listing every service a fix could affect.

  3. ⚡ Gemma Short-Circuit — For common, predictable incidents (flaky third-party integrations, minor infra blips), a lightweight Gemma classifier intercepts and routes to a known remedy — completely bypassing the expensive Gemini RCA. This cuts response time from ~30s to ~3s for routine issues.

  4. 🔧 Applies Remediation — Selects and executes the best course of action:

    • PATCH → generates a fix, opens a real GitHub PR, only after pytest exits 0
    • ROLLBACK → reverts to the last known-good Cloud Run revision via the Admin API
    • RESTART → cycles the Cloud Run instance
    • FLAG_FLIP → trips a Firestore-backed circuit-breaker flag for flaky dependencies

  5. 🔍 Independent Verification Gate — Re-checks the original triggering signal — not the action's own success claim. A patch only counts if the failing test suite now passes. A rollback only counts if the error rate drops below the healthy ceiling. An ambiguous partial improvement is treated as unresolved, never a false success.

  6. 📣 Honest Escalation — If confidence is below a safety threshold, or if all remedies fail, Warden attaches its full investigation trace and escalates to a human. It never takes a destructive guess just to avoid escalating.

  7. 📊 Full Observability — Every step, reasoning trace, and action attempt is persisted to Firestore and surfaced in a real-time Incident Command Center dashboard at the Cloud Run URL.

🛠️ How we built it

Architecture Overview

Tech Stack

  • Google ADK (google-adk==2.7.1): Orchestrates the full reasoning loop via LlmAgent + FunctionTool, with Runner owning production execution.
  • Gemini 3.5 (gemini-3.5-flash-lite primary, gemini-2.5-pro fallback): Root-cause hypothesis, action-class selection, patch generation, plain-English summaries.
  • Gemma 2 (gemma-2-9b-it): Fast first-pass triage classifier that short-circuits the expensive path.
  • Google Cloud Run: Hosts the agent runtime AND is a tool the agent calls back into (rollback/restart). The same platform is both host and target.
  • Google Cloud Pub/Sub: Decouples alert ingestion from agent execution — handles traffic bursts, guarantees delivery, enables async processing.
  • Google Cloud Firestore: The memory bank — stores code_graph, incidents, feature_flags, and agent_runs collections.
  • Google Cloud Build: CI/CD pipeline that runs the test suite on every push via cloudbuild.yaml.
  • PyGithub: Real PR creation for code patches.
  • FastAPI: The Cloud Run entrypoint — webhooks, REST API, static dashboard.

Key Engineering Decisions

  • Deterministic core, ADK shell: orchestrator/loop.py is fully unit-testable without any GCP credentials. The ADK agent wraps this core in production, giving us the best of both worlds — real ADK orchestration in deployment, clean deterministic tests offline.
  • Honest mock layer: Every GCP client resolves its backend from a single WARDEN_MODE switch. Mocks have identical method surfaces to real clients, documented in docs/RISKS.md. The TestRunner (pytest subprocess) is never mocked — the verification gate is always real.
  • Idempotency: utils/idempotency.py claims a webhook delivery_id in Firestore before any work starts. A duplicate delivery is a no-op.

🚧 Challenges we ran into

  • Subprocess environment resolution: Local test runs failed to find system dependencies because they used the wrong Python interpreter. Fixed by targeting sys.executable for all subprocess calls.
  • State separation across mocks: Four GCP clients (Firestore, Cloud Run, GitHub, Pub/Sub) needed to behave as unified singletons — an action's client and its verifier's client must observe each other's writes, just like real GCP. Solved with process-level singleton backends with class-level shared state.
  • Cloud Run CPU throttling: Pub/Sub background processing timed out because Cloud Run throttles CPU when idle. Fixed with --no-cpu-throttling on deployment.
  • Gemini rate limits on free tier: 20 req/day per model. Built a ResilientModel wrapper that cascades from gemini-3.5-flash-lite → gemini-2.5-pro on quota exhaustion.

🏆 What we're proud of

  • 94% test coverage across 80 fully passing unit tests — runnable with pytest -q and zero GCP credentials.
  • A strict Verification Gate that never claims resolution without re-checking the original signal. The agent can say "I don't know" and that's the right answer.
  • Gemma + Gemini dual-model pipeline — a real cost/performance optimization, not just a demo feature.
  • True end-to-end automation: GitHub webhook → Pub/Sub → Triage → RCA → Patch → pytest → PR → Firestore → Dashboard — all without touching a keyboard.

📚 What we learned

An autonomous agent is only as trustworthy as its verification process. It's dangerous to assume an action succeeded because a tool returned status: 200. Warden's core principle: verify the original signal (re-run the exact failing test), not the action's self-report. This one design decision drove every architectural choice.

We also learned that honest escalation is a feature, not a failure. A system that always claims to fix everything is less trustworthy than one that knows when to stop.

🚀 What's next for Warden

  • Multi-language AST parsing (Java, Go, TypeScript)
  • Slack/Teams integration for human-in-the-loop approvals before destructive actions
  • Model Armor integration to filter PII and prompt-injection vectors in open-text logs
  • Agent Registry: publishing Warden as a versioned, discoverable enterprise agent

Built With

Share this project:

Updates

Submission history