SiteMedic

• 

  Dashboard page showing Active incidents

• 

  Dashboard page showing Forecasted incidents

• 

  Compare page

• 

  The Active incident (current status = Remediating, that means Agent is currently implementing the plan)

• 

  Health page

• 

  Settings page

• 

  Dashboard page showing Resolved incidents (approved + rejected incidents)

• 

  Analytics page

• 

  Approval/Rejection options

• 

  Remediation plan

• 

  Audit page

• 

  Remediation idea given by Gemini (live)

• 

  Control Panel

• 

  Finally clicking the Approve and Execute button after the user has verified everything on his own

• 

  Reasoning Trace after triggering an incident

• 

  Final Reasoning trace after Remediation has completed

Inspiration

When production breaks at 3 AM, the slowest part isn't fixing it — it's figuring out what broke. An on-call engineer can burn 20 minutes correlating dashboards, traces, and deploy logs before they even know where to look. I wanted to see if an agent could compress that diagnostic gap: not a chatbot that answers questions, but an agent that investigates, reasons, plans, and acts — with a human always holding the approval switch.

What it does

SiteMedic is an autonomous Site Reliability Engineer. When a production service degrades, it:

1. Detects the incident in real time via the Dynatrace MCP server (webhook-driven, with a polling fallback)
2. Diagnoses root cause — Gemini runs a multi-step ReAct loop, calling Dynatrace and Google Cloud observability tools to investigate problem details, metrics, traces, and recent deployments
3. Reasons transparently — every hypothesis and tool call streams to a live reasoning trace, with a confidence score and explicitly listed uncertainties and alternative explanations
4. Plans a remediation (rollback, scale, restart, Cloud SQL failover, Pub/Sub replay) with an estimated cost impact
5. Waits for human approval — operators preview the exact change via Dry-Run Mode before approving
6. Executes the approved action and verifies the service actually recovered
7. Writes the postmortem automatically — timeline, root cause, and prevention notes

It also forecasts incidents before they breach SLOs, coordinates correlated cross-service incidents into a single cluster, and keeps a tamper-evident audit trail of every decision.

How I built it

• Reasoning: Gemini on Vertex AI — multi-pass diagnosis pipeline (evidence → hypotheses → testing → synthesis)
• Orchestration: Google Cloud Agent Builder coordinating the agent loop and tool calls
• Observability (partner): Dynatrace MCP server as the primary telemetry source — problems, metrics, traces, entities
• Google Cloud observability: Cloud Logging, Cloud Monitoring, and Cloud Trace as a complementary signal source
• Backend: Python + FastAPI on Cloud Run; Firestore for incident state, reasoning traces, and audit log; Secret Manager for credentials
• Frontend: Next.js on Cloud Run — live incident feed, streaming reasoning trace, timeline view, approval UI, analytics dashboard
• Action layer: human-approved remediations executed via the Cloud Run Admin API and gcloud, gated by IAM permission checks

Challenges I ran into

The hardest part was making the agent trustworthy, not just functional. An agent that confidently does the wrong thing in production is worse than no agent. I added confidence scoring with explicit uncertainty, a mandatory human approval gate, dry-run previews of every action, and post-remediation verification that auto-flags failures. I also built a Demo Mode fallback: a TelemetrySource abstraction lets the system replay real captured incident data when the live observability connection is unavailable — so the hosted demo stays reliable while Gemini reasoning still runs live.

Important! Forecasted tab is empty because dynatrace 14 day free trial has ended and hence, the services has stopped as of now. If dynatrace services resume, it will populate automatically — within 15 minutes of Dynatrace coming back online (since prediction interval is 15 min). The prediction loop runs independently and will collect metrics from Dynatrace again, generate predictions, and they'll appear in the Forecasted tab. For the same reason, the Audit page shows errors when it tries to detect loop because it tries to call dynatrace but due to trial expiry services are stopped hence, it gets errors.


Accomplishments I am proud of

• A genuine multi-step agent, not a chatbot — it chains real tool calls to reach a diagnosis
• Honest AI: the agent surfaces what it doesn't know and asks for a human when confidence is low
• Production-grade safety: approval gates, dry-run, IAM checks, verification, tamper-evident audit log
• Graceful degradation — the product works end-to-end even when its primary data source is offline

What I learnt

Building an agent that acts raises the bar far above building one that answers. Every capability had to be paired with a guardrail. The most valuable design decision was keeping a human in the loop by default — it turned a risky autonomous system into a credible assistant an SRE team would actually trust.

What's next for SiteMedic

Wider Google Cloud action coverage (GKE, Memorystore), a learning loop where operator overrides become few-shot examples, multi-agent swarming for large cross-service incidents, and chaos-engineering integration to continuously verify SiteMedic itself and deploying it on multiple codebases to check if it does the job with minimum possible human approvals on its own while still keeping a human in the loop for massive decisions.

Built With

• cloud-logging
• cloud-monitoring
• cloud-run
• cloud-trace
• docker
• dynatrace
• fastapi
• firestore
• gemini
• google-cloud-agent-builder
• mcp
• next.js
• python
• secret-manager
• typescript
• vertex-ai