OpsPilot - Autonomous AI Operations Engineer

Built for the "Build Next-Gen AI Agents - All Things Agentic Hackathon" (Taskmaster Track).

Comment

Inspiration

Modern Site Reliability Engineering (SRE) and cloud infrastructure management are stuck between two extremes:

  1. Rigid, Deterministic CI/CD Scripts: Bash scripts and YAML pipelines execute blind commands sequentially. If a transient error spike occurs, a container pod fails its liveness probe, or latency degrades during a rollout, deterministic scripts cannot reason or self-heal—they either crash or blindly promote buggy code into production.
  2. Text-Only LLM Chatbots: Conversational AI interfaces can explain error codes and draft Kubernetes manifests, but they cannot take autonomous, multi-step action on live infrastructure, monitor telemetry across asynchronous soak periods, or enforce safety gates.

We asked a fundamental question:

What if an operations engineer was an autonomous AI agent that accepts high-level natural language goals, formulates verifiable multi-step execution plans, runs real cloud tools, monitors live telemetry across soak windows, self-heals by rolling back on anomalies, and respects human-in-the-loop authorization gates?

This vision inspired OpsPilot — an autonomous AI operations engineer built for the Taskmaster Track of the Build Next-Gen AI Agents — All Things Agentic Hackathon.

What it does

OpsPilot transforms declarative, high-level operational goals into closed-loop, resilient production actions:

  • Natural Language Goal Planning: SRE operators provide a goal such as: $$\text{Goal} = \text{"Deploy } \texttt{payments-service} \text{ v2.4.0 with 10\% canary, soak for error rate } < 1\%, \text{ and promote to 100\% with approval."}$$ Using Gemini 3.6 Flash, OpsPilot synthesizes an ordered, multi-step dynamic execution plan with explicit risk classifications ($\text{Risk} \in {\text{LOW}, \text{MEDIUM}, \text{HIGH}, \text{CRITICAL}}$).
  • Autonomous Tool Execution via Google ADK: Steps are executed by an agent runtime powered by the Google Agent Development Kit (ADK) using custom FunctionTool implementations for service deployments, smoke test suites, traffic splitting, and workload scaling.
  • Asynchronous Telemetry Soaking & Event Bus: The agent runtime enters a non-blocking WAITING state, subscribing to live telemetry streams on an event bus (backed by Google Cloud Pub/Sub).
  • AI-Driven SLA Evaluation & Autonomous Replanning: When telemetry probes arrive, Gemini evaluates Service Level Indicators (SLIs) against defined objectives (SLOs): $$\text{Error Rate } \varepsilon \le 1.0\%, \quad P_{99} \text{ Latency } \tau \le 200\text{ ms}$$ If a metric anomaly occurs (e.g., $\varepsilon = 4.8\%$), Gemini detects the SLA breach, aborts planned traffic promotions, dynamically generates a remediation plan, executes an emergency rollback to the previous stable release, and restores healthy state with zero human intervention.
  • Human-in-the-Loop Authorization Gates: For destructive or high-risk actions (e.g., promoting a canary revision to 100% traffic or dropping database tables), OpsPilot halts execution in WAITING_APPROVAL state, dispatches automated AI alert notifications, and requires an authenticated operator to authorize before resuming.
  • Real-Time Glassmorphic Cockpit: A dual-theme HUD cockpit designed with Google Stitch MCP provides real-time WebSocket telemetry, live activity log streaming, interactive event simulation, and full operational history replay.

How we built it

OpsPilot is engineered on a modular, production-grade cloud architecture:

graph TD
    A[SRE Operator] -->|High-Level Goal| B[FastAPI & WebSockets Gateway]
    B -->|Synthesize Dynamic Plan| C[Gemini 3.6 Flash via Vertex AI]
    C -->|Ordered Steps & Tool Schemas| D[Google ADK Agent Runtime]
    D -->|Execute Operations| E[Google ADK FunctionTools]
    E -->|Deploy / Scale / Test| F[Target Workload on Cloud Run]
    E -->|Enter WAITING State| G[Event Bus & Cloud Pub/Sub]
    F -->|Telemetry Metrics / Probes| G
    G -->|Wake Agent & Evaluate SLA| C
    D -->|Persist State & History| H[Google Cloud Firestore]
    B -->|Stream Live Logs & Updates| I[Stitch MCP HUD Cockpit]

Core Technologies:

  1. Gemini 3.6 Flash & Google GenAI SDK: Powers dynamic operations planning, risk scoring, metric evaluation, and autonomous incident replanning through Vertex AI and the Gemini API.
  2. Google Agent Development Kit (ADK): Core agent runtime orchestrating LlmAgent, schema generation, tool invocations, and execution loops.
  3. Google Cloud Infrastructure:
    • Google Cloud Run: Managed serverless container deployment hosting the OpsPilot backend and UI with low-latency auto-scaling.
    • Google Cloud Firestore: Persists execution states, versioned dynamic plans, chronological activity logs, and pending approval gates without requiring manual composite indexes.
    • Google Cloud Pub/Sub: Ingests asynchronous operational telemetry signals, health probes, and CI/CD webhooks.
    • Google Cloud Build: Automated container image packaging and continuous deployment pipeline.
  4. Google Stitch MCP & Antigravity IDE: Created the frontend design system from first principles, establishing design tokens, dark HUD/light daylight themes, and CSS spring physics animations.
  5. Backend & Streaming: Python 3.14, FastAPI, and real-time bidirectional WebSockets.

Challenges we ran into

  1. Handling Asynchronous "Waiting" Loops in Agent Runtimes: Standard LLM tool-calling patterns assume immediate synchronous responses. OpsPilot needed a way to yield execution when entering a soak window, persist state to Firestore, and sleep until an asynchronous event wakes the agent up. We solved this by designing custom control tools (wait_for_async_event and request_human_approval) that cleanly suspend and resume execution without blocking server worker threads.
  2. Firestore Composite Indexing Constraints: Firestore queries that combined .where("execution_id", "==", id) with .order_by("timestamp") threw index preconditions in production. We refactored the database layer to query on primary indexed fields and perform fast in-memory sorting and version resolution in Python, eliminating index dependencies.
  3. Real-Time WebSocket Synchronization Across Reconnects: During rapid multi-step deployments, WebSocket message serialisation must handle non-primitive objects (such as datetime and Enums). We implemented custom JSON encoders (model_dump(mode="json")) and a dual-tier WebSocket broadcaster that dispatches updates to both execution channels and global dashboard feeds.

Accomplishments that we're proud of

  • 100% Automated Test Pass Rate: Comprehensive test suite with 17 unit and integration tests covering ADK tools, control flows, multi-step canary execution, soak windows, anomaly replanning, approval gating, and Firestore storage.
  • True Goal-Driven Autonomy: Unlike basic chatbots, OpsPilot accepts an abstract intent and autonomously navigates the entire lifecycle: planning $\rightarrow$ executing $\rightarrow$ soaking $\rightarrow$ evaluating $\rightarrow$ healing $\rightarrow$ gating.
  • Live Production Deployment: Fully deployed and active on Google Cloud Run with verified Gemini 3.6 Flash and Vertex AI integration at: https://opspilot-2mpbafz2ta-uc.a.run.app.
  • Unified Design System with Google Stitch MCP: Crafted an aesthetic, responsive cockpit with custom CSS variables, telemetry sparklines, real-time activity log feeds, and zero bloated frontend dependencies.

What we learned

  • Google ADK Framework Power: Integrating Google ADK's FunctionTool with typed schemas creates structured, safe boundaries around operational tools, making LLM tool selection deterministic and reliable.
  • The Importance of Asynchronous Agent States: Production AI agents cannot operate on synchronous chat request-response cycles alone. Building agents that can sleep, wait for external signals, and resume on event buses is essential for enterprise operations.
  • Designing AI Safety from Day One: By enforcing human authorization gates before high-risk actions ($\text{Risk} = \text{HIGH}$), we proved that autonomous AI agents can operate safely in mission-critical cloud environments without fear of unvetted destructive commands.

What's next for OpsPilot - Autonomous AI Operations Engineer

  1. Multi-Cloud Kubernetes & GKE Integration: Expand Google ADK toolsets to directly interface with Kubernetes APIs and Google Kubernetes Engine (GKE) clusters for pod autoscaling and Canary service mesh traffic splitting via Istio.
  2. Predictive Telemetry Anomaly Detection: Integrate Vertex AI Time-Series Forecasters to predict traffic spikes and memory leaks before they manifest as SLA violations.
  3. Multi-Agent Collaborative SRE Pods: Deploy specialized sub-agents (e.g., Database Migration Specialist, Security Compliance Auditor, Cost Optimizer Agent) orchestrated through Google ADK multi-agent topologies.
  4. ChatOps Integrations: Native bi-directional Slack and Microsoft Teams bot interfaces for approving human gates directly with interactive interactive action cards.

Built With

  • antigravity-ide
  • docker
  • fastapi
  • gemini
  • gemini-3.6-flash
  • google-adk
  • google-cloud-build
  • google-cloud-firestore
  • google-cloud-pubsub
  • google-cloud-run
  • google-stitch-mcp
  • pydantic
  • pytest
  • python
  • vertex-ai
  • websockets
Share this project:

Updates