PatchPilot

AI agent that turns Dynatrace production problems into manageable, tested, human-approved fixes deployed on Cloud Run.

Comment

Inspiration

Every developer knows the 2 a.m. page. An alert fires, and what follows is an hour of glue work: open the dashboard, find the failing service, dig through logs, hunt down the stack trace, open the repo, find the line, write a fix, wait for CI. Observability tools are great at telling you what broke — but the journey from alert to reviewed fix is still manual, repetitive, and done by a tired human.

We wanted to see how far a Gemini-powered agent could compress that loop — from "Dynatrace detected a problem" all the way to "a tested fix is deployed" — without ever taking the human out of the merge decision.

What it does

PatchPilot is an AI root-cause investigator and remediation agent: detect → diagnose → patch → verify.

  • Detect — pulls live problems from Dynatrace using DQL over spans: real exceptions (error spans) and performance regressions (p95 latency over threshold).
  • Diagnose — a Gemini agent queries the Dynatrace MCP server for failing spans and stack traces, reads the actual source code, and pinpoints the root cause to the exact file and line — with an explanation, a confidence score, and alternative hypotheses, streamed step-by-step to the UI as the agent works.
  • Patch — proposes a reviewable unified diff plus a regression test. Human-gated by default: Approve writes the fix to an isolated Git branch (branch-per-fix); merging happens only on an explicit "Confirm fixed". The agent never auto-merges.
  • Verify — opt-in autopilot watches the problem feed, investigates new issues automatically, and coalesces concurrent problems into a single batch: all fixes are applied to the source, then ONE apply → test → build → deploy → verify pipeline runs on Cloud Build — with deployment gated on tests passing, so even autopilot can't ship a broken fix.

Around the core loop: Slack incident digests (re-posted only when the bug set changes), a durable audit history of every patch lifecycle, and a read-only OpenTelemetry instrumentation-gap scanner that proposes missing spans, error records, and attributes.

How we built it

  • Agent — a Go backend built on Google's Agent Development Kit (ADK Go) with Gemini 3 on Vertex AI (gemini-3.5-flash — fast enough for ~30-second live investigations). The agent has three toolsets: the Dynatrace MCP server (20+ tools: execute_dql, spans, problems, entities), read_source (windowed source reading), and propose_patch (records a reviewable diff with rationale).
  • Remediation pipeline — runs on Cloud Build: source snapshots and patch overlays in Cloud Storage, container images in Artifact Registry, deploys to Cloud Run. Build steps stream live into the UI.
  • Frontend — React + TypeScript (Vite): problem list, investigation timeline, diff viewer, pipeline stage tracker, batch panel, settings — all real-time via SSE.
  • Demo "production" service — ShopFlow, an OpenTelemetry-instrumented Go storefront with two seeded bugs (an index-out-of-range crash and a slow report endpoint) exporting real telemetry to Dynatrace.

The whole system — backend, UI, and the MCP server — ships as a single container on Cloud Run.

Challenges we ran into

  • Grounding, not guessing — every agent claim had to come from real telemetry. Getting DQL time windows, Grail retention, and span structures right so stack traces reliably reach the model took real iteration.
  • Two patches, one file — batch auto-patch means multiple diffs can target the same source file. We apply fixes cumulatively, deduplicate by file, then run one pipeline for the whole batch (with per-problem fallback if the batch fails).
  • Building from an ephemeral container — Cloud Run has no Go toolchain or Docker, so the entire apply→test→build→deploy sequence is delegated to Cloud Build using GCS source overlays.
  • Long-running UX — investigations and pipelines take seconds to minutes; everything streams over SSE so the agent's reasoning and the build's progress stay visible.
  • MCP lifecycle — the Dynatrace MCP server runs as a Node subprocess over stdio; supervising it inside a slim production container required careful wiring.

Accomplishments that we're proud of

The full closed loop runs live on Cloud Run: a real Dynatrace exception becomes a root-caused, tested, deployed fix in about five minutes — one click in human-gated mode, or zero clicks with autopilot batching multiple problems into a single pipeline run. And the trust model held throughout: nothing merges or deploys without passing tests and explicit gates. The system also able to know the root cause of the issue and easily apply fixes and deploy immediately either automatically or manually.

What we learned

  • Agents are most reliable with a few narrow, verifiable tools grounded in real data — not many broad ones.
  • Autonomy should be a dial, not a switch: human-approved patches, test-gated auto-remediation, and batch autopilot are the same loop at different trust levels.
  • MCP makes partner integration genuinely plug-and-play — 20+ Dynatrace observability tools for the cost of spawning one server process. -How to efficiently use the google cloud services and AI.

What's next for PatchPilot

Multi-repo and multi-service triage, a pull-request-based flow (GitHub PRs with review comments instead of local branches), and recurring-incident detection that learns fix patterns over time. Test several projects made with different languages programming languages and framework. Add a mobile app client view as an alternative to web view. Push notification on mobile etc.

Built With

Share this project:

Updates