The Canary Whisperer

It's a per-deployment risk simulation agent working with Dynatrace's MCP. It addresses the issue of production crashes and incidents by simulating the error percentage and the blast radius.

Comment

Inspiration

Production incidents are frequently caused by deployments that lacked pre-flight context — not bad code, but insufficient awareness of system state at the time of rollout. CI/CD pipelines validate code correctness but do not assess whether the environment is safe to receive a change. Engineers ship against a health snapshot that is hours old, with no model of which services the feature touches or how fragile those services currently are.

The Dynatrace MCP partner track provided the right integration surface: Dynatrace already maintains a live, typed dependency graph of production services. Connecting that graph to a Gemini agent via the Model Context Protocol made it possible to turn observability data into an actionable deployment decision before a single line hits production.

What it does

Given a plain-English feature description, The Canary Whisperer runs a 4-step analysis pipeline:

  1. Topology mapping — identifies all production services the feature will touch and their downstream dependencies using Dynatrace's Smartscape graph.
  2. Health assessment — fetches incident history (30-day window), current error rates, p99 latency, and SLO compliance for every affected service.
  3. Risk scoring — computes a fragility score per service using a weighted algorithm, then estimates the blast radius (% of users impacted) if the highest-risk service degrades.
  4. GO / NO-GO decision — synthesizes findings into a structured deployment recommendation: GO, GO WITH CONDITIONS (with specific rollout caps, time windows, and monitoring gates), or NO-GO. The output is a full risk report rendered in a React SRE dashboard, and the same pipeline runs as a GitHub Actions bot that auto-comments risk assessments on every pull request. --- ## How we built it Stack: Google ADK · Gemini 2.5 Flash (Vertex AI) · Dynatrace MCP · FastAPI · React + Vite · Docker · Google Cloud Run The system is organized into five layers: | Layer | Path | Responsibility | |---|---|---| | Agent definition | canary-agent/ | Google ADK root agent, Gemini model, MCP toolset | | Business logic | agent/ | 4-step analysis pipeline | | Data integration | dtmcp/ | Dynatrace client (live + mock), Pydantic schemas | | REST API | api/ | FastAPI endpoints | | Frontend | web-ui/ | React 18 + Vite SRE dashboard | Scoring algorithm (agent/analyzer.py): $$\text{fragility} = (S_{\text{inc}} \times 0.40) + (S_{\text{err}} \times 0.25) + (S_{\text{slo}} \times 0.25) + (S_{\text{depth}} \times 0.10)$$ Each sub-score $S \in [0, 1]$. Incident severity is weighted before normalization: $$S_{\text{inc}} = \min!\left(1.0,\ \frac{\displaystyle\sum_{i} w_i}{2.0}\right), \quad w_i \in {1.0,\ 0.7,\ 0.4,\ 0.3,\ 0.2}$$ for AVAILABILITY, ERROR, SLOWDOWN, RESOURCE, CUSTOM respectively. Key architectural decision: The LLM does not drive scoring. Gemini generates the human-readable narrative over the structured output of the Python pipeline — not the decision itself. This makes the system deterministic and auditable. --- ## Challenges we ran into MCP reliability. The stdio MCP transport drops silently under timeout. Any network delay caused the agent to hang indefinitely. This required explicit timeout configuration (30s+) and a dual-mode DynatraceClient with a full mock data layer from the start. Dynatrace OAuth2 scopes. The Dynatrace OAuth2 client credentials flow requires environment-scoped resource URNs in addition to named scopes. Missing scopes return generic 403s with no indication of which scope is absent, requiring iterative cross-referencing against per-endpoint API documentation. Vertex AI auth. Switching from API key auth to Application Default Credentials on Vertex AI requires GOOGLE_CLOUD_PROJECT, GOOGLE_CLOUD_LOCATION, and the LiteLlm model path format (vertex_ai/gemini-2.5-flash) to be configured simultaneously and correctly. A mismatch in any one produces a 404 that is indistinguishable from a missing model error. LLM determinism. Early versions delegated scoring to the model. Results were inconsistent under paraphrase. Moving all structured computation to Python and restricting the LLM to narrative generation resolved this entirely. --- ## Accomplishments that we're proud of
  5. A fully working, end-to-end risk pipeline that takes a plain-English feature description and produces a structured, evidence-backed GO/NO-GO decision in under 10 seconds.
  6. A GitHub Actions bot (/.github/workflows/canary.yaml) that runs the same pipeline on every pull request and posts a risk assessment comment — deployable as a standard CI gate for any team using Dynatrace.
  7. Three fully realized demo scenarios (Safe / Risky / Fatal) backed by realistic mock topology data that reliably reproduce each decision class for evaluation.

- Clean separation between the LLM and the scoring layer, making the system auditable: every score, factor, and decision threshold is deterministic Python code, not model inference.

What we learned

  • ADK prompt contract. Producing consistent, schema-compliant output from Gemini requires the system prompt to define the exact output format, enforce step ordering, and provide a concrete output example. Vague prompts produce structurally inconsistent results.
  • MCP in production. Reliable stdio MCP integration requires explicit connection lifecycle management — timeouts, retry logic, and graceful fallback — not just the happy-path connection.
  • Dynatrace entity model. Dynatrace's Smartscape topology uses typed entity relationships (service → process → host) and a scope-per-resource OAuth model. Understanding this model was essential to building accurate topology traversal.

- Hybrid agent design. Mixing programmatic pipelines with LLMs is more robust than delegating all reasoning to the model. The model excels at language; deterministic code handles classification and scoring.

What's next for The Canary Whisperer

  • Live topology mode — Full end-to-end analysis against a real Dynatrace environment with live incidents and real-time metrics, beyond the current mock data layer.
  • Parallel sub-agents — Split the 4 pipeline steps into independent sub-agents running concurrently, reducing analysis latency from ~8s to ~2s.
  • PR bot as a standard CI gate — Package the GitHub Actions integration as a reusable workflow, configurable with just a Dynatrace environment URL and OAuth credentials.
  • Rollback risk scoring — Extend the model to assess not just forward deployments but rollback safety, given that a rollback to a previous version may re-introduce a previously resolved incident.

Built With

Share this project:

Updates