AegisPilot

## Inspiration

Every engineer who's carried a pager knows the 3am incident: an alert fires, and you're half-awake — grepping logs, squinting at a Grafana dashboard, trying to remember which deploy went out, guessing at a fix, and then writing the postmortem after. It's stressful, slow, and deeply repetitive. The steps almost never change.

We wanted to see if a team of AI agents could actually run that loop — not just chat about it, but take real action, safely, with a human in control. So we built AegisOps: an autonomous SRE on-call agent that triages, diagnoses, remediates, and learns.

## What it does

When a production alert fires, AegisPilot runs the entire on-call loop autonomously:

Triage — classifies severity (SEV1–4), the affected service, and blast radius.
Diagnosis — pulls and classifies the logs, and reads the actual Grafana dashboard image with Gemini vision to confirm the anomaly (with an AI-generated annotation overlaid in
the UI).
Correlation — scores recent deploys by how close they landed to the incident and names the probable root cause with a calibrated confidence.
Memory — a vector-similarity search over past incidents: "seen before, resolved by rollback in ~4 minutes."
Remediation — proposes a reversible fix and halts for human approval. The destructive action is never something the model can trigger on its own.
Comms — on approval, it executes the fix, auto-writes a full RCA with Gemini Pro, posts a PII-scrubbed incident card to Slack, and files a ticket.

Two things make it feel like a real product, not a canned demo:

Bring your own incident — a judge can type in their own service, paste real log lines, add a recent deploy, even upload a dashboard screenshot, and the real agents diagnose
exactly that. Nothing is randomized.
It learns — every resolved incident is written back into memory, so the next time a similar one appears, the Memory agent instantly recalls the known fix.

Every agent step — reasoning, tool calls, tokens, latency — streams live to an "Incident War Room" and is persisted to a full audit trail.

## How we built it

Orchestration: Google ADK (LlmAgent + Runner). Six specialized agents, each with its own instruction, tools, and model, coordinated through a state machine (DETECTED → TRIAGED → DIAGNOSED → CORRELATED → AWAITING_APPROVAL → REMEDIATING → RESOLVED).
Models: Gemini 3.5 Flash for the five fast agents and Gemini Pro for the RCA, all on Vertex AI via Application Default Credentials (no API keys). Every model call —
Flash and Pro — is wrapped in a 503-safe exponential-backoff-with-jitter retry.
State & memory: Firestore (incidents, deploys, logs, incident_memory, agent_registry, audit_log).
Event ingestion: Pub/Sub — alerts arrive on a topic and a push subscription drives the orchestrator.
Serving: a single container (FastAPI + a React + Vite + Tailwind war room) on Cloud Run, with Server-Sent Events streaming every reasoning step in real time.
Governance: an async approval gate, a self-implemented PII scrubber for anything bound for Slack, and an agent registry — all shown in the UI.
Cloud-portable by design: StorageService and EventBus interfaces mean the same code runs locally on SQLite + an in-process bus, or on Firestore + Pub/Sub, by flipping one env
var.

## Challenges we ran into

Reliability on a throttling backend. Vertex 503s under load, so we funneled every call through one retry wrapper — including ADK's internal model calls, via a custom BaseLlm
subclass — so a live demo can't crash on a blip.
Human-in-the-loop on serverless. The approval gate pauses the run for a human, and the pipeline continues as a background task after the Pub/Sub push returns. On Cloud Run that
meant enabling CPU-always-allocated so the async, gated work keeps running between requests.
Keeping the AI honest. We ground every agent in deterministic tools and reconcile the model's output against hard signals — confidence is capped to the deploy-proximity evidence, memory has a similarity floor, and severity has a rubric the model can't soften. When signal is weak, it says "novel incident" instead of hallucinating.
Model availability. gemini-3.5-flash is published in Vertex's global region, not us-central1, so we split the model region from the compute region.
Reproducible builds. Loose dependency pins made container builds backtrack for minutes; pinning the exact verified versions fixed it.

## Accomplishments that we're proud of

It's real — real ADK agents, real Vertex Gemini calls (you can see per-step token counts and latencies in the audit trail), fully deployed on Cloud Run, end to end.
Multimodal that matters — Gemini vision reading the actual dashboard, not just parsing text.
Governance built in — a physical approval gate and PII scrubbing, not an afterthought.
It genuinely learns — a novel incident is flagged as novel, and after it resolves, a repeat is recalled at near-perfect similarity with the known fix.
Judges can drive it with their own data — and it degrades honestly on sparse input, which we think is more trustworthy than false confidence.

## What we learned

Agent reliability is mostly about grounding and reconciliation — LLMs are far more trustworthy when they reason over real structured data and their outputs are checked against
deterministic signals.
The interesting engineering in agentic systems is often the plumbing: retries, streaming observability, human-in-the-loop pauses, and how those interact with a serverless runtime.
A system that knows what it doesn't know ("no strong prior," low confidence) earns more trust than one that always sounds certain.

## What's next for AegisPilot

Real executors — swap the clean, clearly-labeled simulation for real, gated actions via GitHub Actions / Argo Rollouts / kubectl.
Real signal sources — ingest from Prometheus/Grafana, Cloud Logging, and PagerDuty/Opsgenie.
Per-team memory & policies — org-specific runbooks, and auto-approval policies for low-risk, high-confidence fixes.
An evaluation harness — replay historical incidents to measure diagnosis accuracy and time-to-resolution over time.