Spark - SRE agent

SPARK cuts incident resolution from 45 minutes to 60 seconds : Gemini diagnoses -> you approve -> GitLab rolls back automatically.

Comment

Inspiration

At 2 AM, when production goes down, no engineer should spend 45 minutes manually reading logs they barely remember how to navigate. That's the reality of on-call SRE work today groggy context-switching, Slack war rooms, runbook archaeology.

The question I couldn't stop thinking about: what if the investigation happened automatically, and the engineer only had to make one decision?

That's SPARK.

What it does

SPARK is an autonomous incident response agent. When Dynatrace fires an alert:

  1. Detects — Dynatrace MCP server surfaces the problem: severity, affected entities, evidence
  2. Diagnoses — Gemini 2.5 Flash on Vertex AI calls 4 tools in sequence:
    • get_problem -> problem details and affected services
    • query_metrics -> memory, latency, CPU time-series
    • search_logs -> application logs around the incident window
    • get_recent_deployments -> GitLab pipeline history for the past 3 hours
  3. Correlates — cross-references deployment timestamp with anomaly start time
  4. Briefs — posts a single-screen summary: root cause, confidence score (0–100%), rollback target
  5. Notifies — sends an approval card to Slack (#all-spark) with one-tap Approve / Reject
  6. Executes — on approval, triggers the GitLab rollback pipeline automatically

Total time: alert → rollback in under 60 seconds. Human always in the loop. Always.

How I built it

Tech Stack

AI Agent

  • Google ADK
  • Gemini 2.5 Flash
  • Vertex AI

Observability

  • Dynatrace MCP

CI/CD

  • GitLab REST API

Infrastructure

  • Google Cloud Run

Notifications

  • Slack Block Kit

Frontend

  • React + Vite

Architecture Flow

  1. Dynatrace detects an incident.
  2. Cloud Run webhook receives the alert.
  3. Google ADK launches Gemini 2.5 Flash.
  4. Agent gathers:
    • Problem details
    • Metrics
    • Logs
    • Deployment history
  5. SPARK generates a root-cause briefing.
  6. Slack approval card is sent.
  7. Engineer approves or rejects.
  8. GitLab rollback executes automatically.
  9. Resolution and post-mortem are recorded.

Challenges I ran into

1. Multi-worker memory isolation — Gunicorn with 2 workers meant sessions created in Worker 1 were invisible to Worker 2. Every poll request potentially hit a different process. Demo appeared broken. Fixed by switching to --workers 1 --threads 8.

2. Dynatrace MCP in a headless Cloud Run environment — The MCP server uses browser OAuth by default. We built a fallback chain: MCP → OAuth2 client credentials → realistic mock data, so the agent works end-to-end in all environments.

3. Slack interactive buttons — Slack requires a registered Interactivity Request URL to process button clicks. Built a direct URL-action fallback (/approve/<incident_id>) so buttons work immediately without additional Slack App configuration.

4. The Foursquare scenario — Most agents would see an incident and immediately recommend a rollback. Foursquare's 2010 MongoDB OOM had no recent deploy — the root cause was purely a capacity issue. Getting the agent to correctly say "don't rollback, fix the index" required careful prompt design around the correlation logic.

Real-world benchmark: 5 actual production post-mortems

We tested against incidents from github.com/danluu/post-mortemslink — real failures at real companies, with real resolution times:

Benchmark Results:

rust-lang

  • Human MTTR: 13 min
  • SPARK: 16 sec
  • Score: 100%

Cloudflare

  • Human MTTR: ~60 min
  • SPARK: 22 sec
  • Score: 87%

PagerDuty

  • Human MTTR: ~30 min
  • SPARK: 19 sec
  • Score: 86%

Foursquare

  • Human MTTR: 17 hrs
  • SPARK: 38 sec
  • Score: 77%

CircleCI

  • Human MTTR: Hours
  • SPARK: 35 sec
  • Score: 52%

Average: 80% across 5 incidents.

Average: 80% across 5 real incidents. 5/5 correct rollback decisions.

The Foursquare result is the one i am most proud of: the agent correctly identified that no deployment caused the outage and recommended scaling infrastructure instead of rolling back the same conclusion human engineers took 17 hours to reach.

$$\text{MTTR reduction} = \frac{45\text{ min} - 0.5\text{ min}}{45\text{ min}} \approx 98.9\%$$

What I learned

  • Tool sequencing matters more than model size. The agent's accuracy came from the order of tool calls — problem details first, then metrics, then logs, then deployments — not from a larger model. The correlation logic is in the prompt, not the weights.

  • Human-in-the-loop is a feature, not a limitation. Every auto-remediation system eventually causes a worse incident by acting on a misdiagnosis. Keeping the engineer as the final gate — with all context pre-loaded — is the right design for production.

  • MCP changes what's possible for observability agents. Before MCP, integrating Dynatrace meant building and maintaining a custom REST wrapper. With the @dynatrace-oss/dynatrace-mcp-server, the agent gets structured, schema-defined access to problems, metrics, and logs with zero glue code.

  • The hardest incidents don't have a deploy to blame. Building for the Foursquare scenario forced us to make the agent reason about absence of evidence — a much harder problem than correlating timestamps.

What's next for SPARK — SRE Agent

  • Multi-service incident graphs — when an alert spans 3 microservices, trace the dependency chain and identify the blast radius before recommending a fix
  • Learning from decisions — store engineer approval/rejection patterns to improve confidence calibration over time
  • Proactive alerting — run the diagnosis loop on anomaly precursors before Dynatrace fires a P1, giving engineers a 10-minute head start
  • Expand beyond rollbacks — scale operations, config patching, feature flag toggling as additional approved action types

Built With

  • dynatrace-mcp
  • gemini-2.5-flash
  • gitlab-ci-cd
  • google-cloud
  • google-cloud-agent-builder
  • python
  • react-js
  • vertex-ai
Share this project:

Updates