SpecterOps — Autonomous Post-Incident Forensics

Four autonomous agents turn a live Dynatrace alert into an evidence-backed post-mortem in under 60s — Gemini on Vertex AI + the official Dynatrace MCP server. No human in the loop.

Comment

🌙 Inspiration

Every on-call engineer knows the 3am phone scream. Production is down — and the first hour isn't fixing anything. It's detective work: pivoting between traces, logs, metrics, and dashboards just to answer one question — what broke, and why?

Industry studies put mean-time-to-resolution for serious incidents in the hours, and for a real business every one of those minutes is lost customers and lost revenue. Worst of all, a human being is sitting in the dark, exhausted, doing pattern-matching over telemetry that a machine should be doing.

That investigation is an agent problem. So we built the agent.

🤖 What it does

SpecterOps takes a live Dynatrace problem and runs the entire root-cause investigation autonomously — no human in the loop — streaming every step live to a dashboard and delivering a finished post-mortem in under 60 seconds.

It's a pipeline of four specialized agents:

Agent Job
🛡️ SentinelAgent Classifies the alert; extracts the affected service, severity, and an investigation plan
🔍 TraceArchaeologist Pulls live problems, logs, metrics & entity topology from Dynatrace in parallel and rebuilds the incident timeline
🎯 BlameMapper Reasons over the timeline to build a directed causal graph and pin the single root cause with a confidence score
📝 NarratorAgent Writes a production-grade, 10-section post-mortem — impact, timeline, root cause, and the exact one-line fix

The result is visualized with a real-time agent pipeline, a D3 force-directed causal graph, and a live "Time to Diagnose" counter — and the finished RCA can be pushed straight back onto the Dynatrace problem as a comment.

In the reference incident, SpecterOps traces a 34% error-rate outage back to a single missing database index — through a circuit-breaker cascade across three services — and writes the fix:

CREATE INDEX CONCURRENTLY idx_payment_methods_user_id ON payment_methods (user_id);

🛠️ How we built it

  • Reasoning — Google Gemini 2.5 Flash on Vertex AI (google-genai SDK), using Cloud Run's own service identity (no API keys), with a structured-JSON contract and a graceful model-fallback chain.
  • Dynatrace — the heart of the track — SpecterOps is a Model Context Protocol client that connects to the official @dynatrace-oss/dynatrace-mcp-server and calls real tools — list_problems, execute_dql (Grail DQL), and find_entity_by_name — against a live tenant. A single MCP session is spawned and reused, so login happens once.
  • Backend — Python · FastAPI · asyncio · Server-Sent Events, on Google Cloud Run (Docker, with Node + the MCP server baked into the image).
  • Frontend — React 18 + Vite · Tailwind · Framer Motion · Three.js (3D hero) · D3 (causal graph), on Firebase Hosting + Auth.
  • Persistence — Google Cloud Firestore, authenticated by the Cloud Run service identity, so investigation history survives restarts.

🧭 Data sources

Live Dynatrace observability data — Davis problems, Grail logs and events, metrics, and monitored-entity topology — pulled through the MCP server locally and the Dynatrace REST API on the hosted deployment. A deterministic simulation layer powers four built-in failure scenarios so the product also runs end-to-end with zero accounts.

🧱 Challenges we ran into

The most interesting challenge was authentication topology. The Dynatrace MCP server authenticates through an interactive browser OAuth flow — perfect on a laptop, impossible on a headless Cloud Run instance with no browser and no human to approve the login.

Rather than fake it, we made the agent path-aware: it speaks MCP locally — the way an MCP server is designed to run, right next to the developer/agent — and falls back to Dynatrace's token-based REST API when deployed headlessly. Same agent, same tenant, the correct transport for each environment. We also normalized Dynatrace units (microseconds, percentages) and chose to hide anything unresolvable rather than show it wrong.

🏆 Accomplishments that we're proud of

A genuinely autonomous, multi-agent system — not a chatbot wrapper — that turns live telemetry into a real causal graph and a publishable post-mortem in under a minute. It's deployed as a polished, business-grade product on a 100% Google Cloud + Dynatrace stack, with graceful degradation everywhere so it never breaks on stage — and it runs with zero accounts when you want it to.

📚 What we learned

  • MCP is a local-first protocol by design — understanding why led to a cleaner architecture than forcing it where it didn't belong.
  • Running Gemini on Vertex AI via Cloud Run's service identity is dramatically simpler and safer than shipping API keys.
  • A strict structured-output contract + deterministic fallbacks is what makes an LLM agent demo-reliable under pressure.

🚀 What's next for SpecterOps — Autonomous Post-Incident Forensics

  • Approval-gated auto-remediation — wire the proposed fix straight into CI/CD.
  • Per-user, multi-tenant Dynatrace connect via OAuth client credentials — which also unlocks fully headless MCP on the hosted deployment.
  • Auto-file a GitHub / Jira ticket from the finished post-mortem, closing the loop from alert to tracked remediation.

Built With

  • asyncio
  • cloud-firestore
  • cloud-run
  • d3.js
  • docker
  • dynatrace
  • dynatrace-mcp-server
  • fastapi
  • firebase-auth
  • firebase-hosting
  • framer-motion
  • gemini
  • gemini-2.5-flash
  • github-actions
  • google-cloud
  • google-genai
  • grail-dql
  • javascript
  • mcp
  • model-context-protocol
  • node.js
  • pydantic
  • pytest
  • python
  • react
  • react-three-fiber
  • server-sent-events
  • tailwindcss
  • three.js
  • uvicorn
  • vertex-ai
  • vite
Share this project:

Updates