Incident Command Center

• 

  demo view

• 

  demo view2

Inspiration

We do enterprise integration support for enterprise Oracle Fusion ERP / Integration Cloud platforms — dozens of integrations moving sales orders, shipments, POs and EDI between Fusion and warehouse, WMS(warehouse management systems) and supplier systems. The pages we get called about most start with the same sentence: "there's no data showing." A dashboard is blank, a warehouse says it hasn't received shipments since 2am, and nobody's sure yet whether that's one broken integration or six.

The part that burns the clock is never the fix. It's scoping four correlated zero-count alerts into one incident; bisecting which of last night's changes did it; reading a CLOB error column out of a log table; remembering that a concurrency flag gets stuck exactly like this when a scheduled run crashes; and then writing the status three ways — for the client, for the bridge, for the delivery lead. Every one of those is a well-defined task. That's a team of agents.

What it does

A SequentialAgent pipeline of five specialists, each handing the next a structured finding:

1. Detection Agent — reads the dashboard Process Counts, the firing monitors, and the integration dependency map, and collapses the correlated zero-count alerts into one scoped incident: the primary integration, its blast radius, a severity, when it started (from the metric inflection, not the alert time), and what's confirmed still healthy so the scope is bounded.
2. Root Cause Agent — checks the concurrency gate first, reads the expanded CLOB error, pulls the run trace showing the crash landed before the flag reset, correlates it with the change activated 90 minutes earlier, and explicitly rules out the innocents (the on-prem agent, the BIP endpoint, the database).
3. Knowledge Agent — matches the applicable runbooks and searches institutional memory for the closest prior incident, returning its resolution and its MTTR.
4. Resolution Agent — produces the plan in the order that matters: remove the cause first (clearing the gate before rolling back just re-sticks it), then the exact SQL to clear it with a watermark rewind for backfill, then an ad-hoc run, then verification thresholds, a rollback-of-the-rollback, and an ETA anchored to the precedent.
5. Communication Agent — writes the client status update (no internal names, no blame), the internal Slack update, and the exec brief, then raises an Ivanti P2 and records the postmortem.

The Knowledge Agent's memory searches a history file plus every postmortem the system has written; the Communication Agent writes that postmortem at the end of every run — so the next similar incident opens with this run's hindsight already retrieved. All 14 tools are also an MCP server: demo_mcp_triage.py walks the whole read-then-generate flow — down to producing the Ivanti ticket and the postmortem — with no model in the loop.

How we built it

• Google ADK, SequentialAgent as the root — Pillar 1 of the ADK codelab: the graph is fixed and drawable before any input, because incident response is a pipeline. State flows agent-to-agent via output_key and {key?} instruction templating, kept optional so each agent still runs standalone in the dev UI.
• Gemini 3.5 Flash on Agent Runtime
• A realistic seeded scenario — seed/ is a self-consistent incident: matching alerts, metrics, a distributed trace, a dependency graph, two runbooks, and four prior incidents (one a near-exact match). It's a diagnostic problem — the obvious suspects are all healthy, so root cause has to read the gate, the trace, and the change record. Every tool is a thin reader over a seed file; each is a one-function swap to a real Oracle ATP query / OIC monitoring call / BIP SOAP call.
• MCP via FastMCP — the 14 tools plus an icc://alerts resource; demo_mcp_triage.py is a standalone MCP client.
• A custom BaseMemoryService doing keyword-overlap retrieval over the history JSONL plus the written postmortems — small enough to cat, adapted from the MonsterTix memory codelab.
• Eval — an ADK evalset pinning the full 13-tool trajectory across all five agents and checking the Ivanti ticket + postmortem get written, plus deterministic pytest for the tools and the memory ranking.
• A live dashboard — the five stages fill in left to right as the pipeline runs, driven off each event's author; the rail shows the flatlined Process Counts and the stuck flag in red; /dev-ui is one click away.

Challenges we ran into

• Making the scenario a diagnosis, not a lookup. The first version had the error obvious. Keeping the on-prem agent, the BIP endpoint, the database and an unrelated integration all healthy forces the Root Cause Agent to actually read the evidence and rule the innocents out — and that "ruled out" list is the most convincing part of the demo.
• Getting the fix order right. Clearing the concurrency flag before rolling back the bad change just re-runs the failure and re-sticks it. The Resolution Agent's instruction had to make the ordering explicit and explain why, or the LLM proposes the tempting one-step fix.
• State handoff across five agents. output_key plus naming the upstream keys explicitly, made optional so a single agent is still runnable alone.
• Eval determinism across five agents. Thirteen tool calls; the LLM sometimes reorders two of them. The evalset uses a lenient trajectory score and the hard guarantees live in the model-free tests.
• Vertex quota. The five-agent bridge bursts ~15–20 calls; a rate-limited project returns 429. Mitigated with a lighter model, not code changes.
• Keeping it generic. The scenario models a real failure class with no client, product, or engagement names anywhere in the repo.

Accomplishments that we're proud of

• One trigger, a whole incident worked. From a single alert to a scoped incident, a root cause with an evidence chain, an ordered fix, three drafted comms, a ticket, and a postmortem.
• The evidence chain and the ruled-out list. An agent that says "it's the change" is a guess; one that says "it's this change, here are the three log lines and the trace, and it's not the database because connections are flat" is a colleague.
• The self-improvement loop actually closes. Run a similar incident again and the Knowledge Agent opens with the previous writeup already in hand.
• The whole capability runs with no model through the MCP tools — useful for a runbook automation, not just a demo.

What we learned

• A fixed SequentialAgent is underrated. Incident response doesn't need dynamic orchestration — it needs five agents that are each really good at one handoff.
• Memory you can read beats memory you have to trust — especially when the demo is about the system getting faster every time it's used.
• Building the tools as an MCP server first made the agent one client among several, and made the deterministic demo possible.
• The ordering of a remediation plan is itself expertise worth encoding.

What's next for Multi-Agent Incident Command Center

• Real connectors — swap the seed readers for Oracle ATP, the OIC monitoring REST API, and a real ITSM, behind the same tool signatures.
• Live trigger — a webhook / Pub/Sub entry point so a real alert convenes the bridge automatically (the MonsterTix scheduled-agent pattern).
• Human checkpoints — pause before Resolution executes anything, and let an on-call approve or edit the plan (a task-mode agent).
• More failure classes — third-party outage with a circuit breaker, an on-prem agent down, a schema drift, a queue backup — each with its own runbook and precedent.
• Postmortem analytics — mine the accumulated postmortems for recurring causes and propose the systemic fix.

Built With

• adk-eval
• agent-development-kit
• cloudrun
• fastapi
• fastmcp
• gemini-3.6-flash
• google-adk
• mcp
• oracle-fusion
• oracle-integration-cloud
• python
• uvicorn