Inspiration
Every engineer who has been on-call knows the 2 AM scramble: a Dynatrace alert fires, Slack lights up, and the next 30 minutes are spent answering the same three questions — What is broken? Why? What do I do? — while the clock runs and users suffer. The question that inspired Incident Commander was simple: what if the first three steps of an incident happened autonomously, in seconds, and the human only had to click one button?
The Google Cloud Rapid Agent Hackathon's Dynatrace track gave me the perfect canvas. Dynatrace already has the best situational awareness in the room — Davis AI, Grail DQL, topology context, and a brand-new MCP server that exposes all of it to an LLM agent as structured tool calls. Pairing that with Gemini 3's reasoning and Google ADK's agent framework felt like a natural fit for a system that could genuinely close the loop: detect → diagnose → propose → approve → act → verify.
What I Built
Incident Commander is a full detect–diagnose–propose–approve–remediate–verify loop:
A "patient" microservice (an Orders API on Cloud Run, instrumented with OpenTelemetry) that ships traces, metrics, and logs to Dynatrace. It exposes a
/breakendpoint that injects one of three faults on demand.An Incident Commander agent (Gemini 3 via Google ADK) that watches for open Davis problems through the Dynatrace MCP server. When a problem appears, it pulls the Davis narrative, Grail DQL signals, and raw service state — then emits a structured
Diagnosis: root cause, evidence citations, and exactly one proposed remediation.A human-approval gate — the agent names an action; a deterministic orchestrator awaits a human click before anything executes.
A war-room web UI (two-pane, vanilla JS, no build step) that streams the agent's live reasoning on the right and shows system health on the left. One Approve button fires the fix; one Reject button closes the loop silently.
Recovery verification against the live service — the agent doesn't declare victory until
/healthconfirmsactive_faults == [].
The architecture deliberately separates concerns. The LLM's only job is classification-with-citations: read signals, choose one of three labels, justify the choice. The orchestrator owns the label→function dispatch table, so the LLM can never trigger an action it wasn't explicitly taught. This is the safety guarantee: the Gemini step is a bounded classifier, not a free-form executor.
How I Built It
The stack is intentionally lean:
| Layer | Technology |
|---|---|
| Patient app | Python · FastAPI · OpenTelemetry |
| Agent framework | Google ADK (LlmAgent, output_schema) |
| LLM | Gemini 3 (Vertex AI) |
| Observability | Dynatrace MCP server (npx @dynatrace-oss/dynatrace-mcp-server) |
| Structured DQL | Dynatrace Platform API (fetch dt.davis.problems) |
| Deployment | Google Cloud Run (two services, one image) |
| UI | Vanilla JS · Server-Sent Events |
The observability layer is a swappable ObservabilitySource protocol with two implementations: LocalObservabilitySource (reads the patient directly — fast for demos) and DynatraceObservabilitySource (reads real Davis problems via Grail DQL — the real-integration story). The commander only imports the interface; swapping sources is one environment variable (OBSERVABILITY_SOURCE=dynatrace).
Reliable structured output was a core engineering concern. ADK's output_schema forces Gemini to return schema-valid JSON (a Pydantic Diagnosis model) rather than prose. The orchestrator validates the output — proposed_remediation must be in the allowed set, remediation_args must match the tool's signature — and feeds validation errors back to the model for one bounded retry before surfacing an error state. The user is never shown an un-actionable proposal.
For the latency fault, the relationship between capacity, load, and delay follows a simple model the agent reasons about:
$$\text{delay_ms} = \max(0,\ (\text{load_level} - \text{capacity}) \times 10)$$
Setting $\text{capacity} \geq \text{load_level}$ drives the delay to zero. The agent reads both values from /admin/state and proposes the exact scale_service(capacity=load_level) call — no guessing.
What I Learned
Hybrid observability design. Real-world detection (Davis) and immediate recovery verification (your own service's /health) serve different purposes. Davis is authoritative and rich but minutes-scale; the patient's /health endpoint is milliseconds-scale. The hybrid design — Davis for eyes, patient endpoints for hands and confirmation — gave me both depth and speed.
The approval gate as a first-class design element. Most agent demos either skip human oversight or bolt it on. Designing the gate as a plain asyncio.Future — resolved by /approve, cancelled by /reject, with no LLM in the wait — made it deterministic, testable, and safe. It's the clearest possible illustration of "the agent proposes, the human decides."
Constrain the LLM's scope ruthlessly. Giving Gemini a tight instruction ("read these three signals, choose one of three labels, justify your choice") produced reliable, consistent Diagnosis objects. A broader instruction ("figure out what's wrong and fix it") produced prose, hedging, and occasional hallucinated tool calls. Narrow scope + output_schema = a production-worthy signal.
ADK's MCP toolset bridges worlds cleanly. McpToolset with StdioConnectionParams turns the Dynatrace MCP server (a Node.js process) into first-class ADK tools with zero glue code. The agent calls query_problems and get_problem_details the same way it would call any Python function.
Challenges
Davis detection lag vs. demo pace. Dynatrace's AI detection is minutes-scale and traffic-dependent — not ideal for a 3-minute demo. The solution was the ObservabilitySource abstraction: local mode gives an instant feedback loop for filming; dynatrace mode is the real integration story for the judges. A single env var switches between them with no code change.
Structured output reliability. Early prompts produced Gemini responses that mixed JSON with prose commentary. Switching to ADK's output_schema (Pydantic model enforcement) plus a validation-then-retry loop raised reliability to near-100% across the three fault types.
Cloud Run and the background poller. The commander's autonomous polling loop is a background asyncio task. Cloud Run's default CPU throttling freezes background tasks between requests — the poller would never fire. The fix was deploying the commander with --no-cpu-throttling --min-instances=1, keeping one instance always-on. This trades a small standing cost for a reliably autonomous demo.
Keeping the "real vs. modeled" line honest. The Gemini diagnosis, the HTTP remediations, the recovery verification, and the Dynatrace MCP integration are all real. The patient's faults are injected, not organic. Being explicit about this distinction in the README — and in the demo — is what makes the project credible rather than just clever.
Log in or sign up for Devpost to join the conversation.