Autonomous Incident Resolver Agent

• 

  An agent that investigates, decides, and acts — safely, gated by a deterministic safety policy engine

• 

  Observe, diagnose, decide, act & verify — four steps, fully visible, with the Safety Engine as a distinct decision point

• 

  Overview: active/resolved incidents, auto-fix rate, and average recovery time at a glance

• 

  Four reproducible failure scenarios, including a deliberately blocked one (credential exposure)

• 

  A live incident awaiting human approval before any fix runs

• 

  Root cause, confidence, and the proposed fix — with Approve/Reject in the human's hands

• 

  Full explainability: root cause confidence, historical success rate, and every tool the agent called

• 

  Once approved, the fix executes, health is re-verified, and the result — including who approved it — is permanently recorded

• 

  Incident memory: past occurrences of the same root cause, including similarity score and outcome

• 

  What the agent has learned: root cause categories and per-action remediation success rates

• 

  Full safety controls: autonomy mode, a global kill switch, and the action risk whitelist the policy engine actually enforces

• 

  Per-service risk overrides — a CRITICAL production service can require stricter approval than the global default

• 

  Auto-generated postmortem, grounded strictly in the incident's own recorded timeline

Inspiration

Most "autonomous AI agent" demos are chatbots that write text describing what they'd do. We wanted to build one that actually does something — and, just as importantly, knows when not to. Production incident response is a perfect proving ground: it's high-stakes, action-oriented, and the cost of a wrong autonomous decision is real.

What it does

The Autonomous Incident Resolver watches a live target service. When it breaks, the agent runs a closed loop: observe → diagnose → decide → act → verify → re-plan. Gemini 3.5 Flash (via Google ADK) reads logs, inspects config, and proposes a root cause and a fix. But the agent's own opinion of that fix's risk is never trusted — a separate, deterministic Safety Policy Engine is the only thing that decides whether it actually runs:

• LOW risk → executes automatically
• MEDIUM risk → waits for human approval
• HIGH risk / unknown action → never executed, escalated to a human — no exceptions

We prove this live with a leaked-credential scenario: the agent correctly diagnoses the problem and proposes rotating the credential — the right fix. The Safety Policy Engine blocks it anyway, because that action isn't on the safe-execute whitelist, and escalates to a human instead. The fix is never attempted. That's not a bug — it's the entire point.

Everything is explorable in a live operations dashboard: full audit trail (who approved what and why), incident memory (similarity search against past successes and failures), autonomy controls (Observe Only → Autonomous Low-Risk), a global kill switch, and auto-generated postmortems.

How we built it

• Gemini 3.5 Flash (Vertex AI) via Google ADK's LlmAgent, wired with read-only tools (logs, config, error-pattern extraction) plus one required output tool: propose_remediation.
• A deterministic Safety Policy Engine (app/agent/policy.py) — a base action risk whitelist, per-service risk profiles, a global autonomy mode, a kill switch, and an execution rate limiter. This is the only place an execution decision is made; the orchestrator never checks risk itself.
• FastAPI backend on Cloud Run running the full state machine — investigate, evaluate policy, apply, verify, re-plan on failure (capped attempts), escalate.
• Firestore for incidents, remediations, the activity/audit log, settings, and service risk profiles.
• A simulated target service, also on Cloud Run, with four reproducible failure scenarios so the loop is demoable and judge-repeatable.
• Next.js dashboard with Firebase Auth (Google sign-in or guest mode) — guests get full read/write access with no signup, so a judge can try the live demo in seconds.

Challenges we ran into

• Never letting the LLM grade its own risk. It was tempting to have the model self-report a risk level alongside its proposal. We deliberately never wire that field into any execution decision — the whitelist and policy engine are the only source of truth, because a model's stated confidence and its actual risk are not the same thing.
• Concurrency and re-planning correctness. A remediation that fails verification has to re-plan without repeating the same failed action, cap out after a fixed number of attempts, and never let two concurrent requests approve/reject the same remediation twice. Several rounds of dedicated adversarial testing (not just happy-path testing) surfaced real bugs here — including a cross-incident authorization bug and unordered Firestore queries silently dropping recent data past a scale limit — all fixed with regression tests.
• Keeping the demo reproducible for judges without a real production environment to point at, while still proving every claim live rather than by inspection.

Accomplishments that we're proud of

• A safety architecture where the LLM's opinion of risk is structurally irrelevant to what executes — provable live, not just by reading code.
• 117 automated tests, including adversarial coverage most hackathon projects skip entirely: concurrency races, malformed LLM output, authorization boundaries, and Firestore partial failures.
• A full audit trail and incident-memory system that make the agent's reasoning inspectable end to end, not a black box.

What we learned

The safety story is the product, not a feature bullet. The most convincing thing we can show a judge isn't "the agent fixed something" — plenty of agent demos do that. It's "the agent correctly figured out the dangerous fix, and a system outside the model's control refused to run it anyway." That single scenario does more to prove trustworthy autonomy than any amount of successful auto-healing.

What's next for Autonomous Incident Resolver Agent

• A dedicated demo button for LOW-risk auto-execution (currently exercised by the test suite and reachable via the API, but not wired to a button).
• Closing the narrow multi-instance approve/reject race window with a Firestore transaction.
• Real infrastructure integrations (real log sources, real deployment rollback APIs) beyond the simulated target service.

Built With

• fastapi
• firebase-auth
• firestore
• gemini
• google-adk
• google-cloud-run
• next.js
• python
• react
• tailwind-css
• typescript
• vertex-ai