Inspiration
Every on-call engineer knows the odd hours ritual. Phones fires. Open Dynatrace. Scroll through several services. Click a problem - 200 spans, all red. Open Cloud Logging in another tab. Search severity>=ERROR. 12,000 results. Open Slack. #incidents has 83 new messages.
By the time you mentally correlate the timeout in Dynatrace with the connection pool exhaustion in Cloud Logging, 45 minutes have passed. The incident escalated. Users would have noticed.
The problem isn't lack of data. It's too much data, scattered across too many tools, waiting for a human to become the integration layer. We built OnCall Agent because we are tired of being the glue between our own observability stack.
What it does
OnCall Agent is an autonomous investigation engine that receives Dynatrace alerts, fetches evidence across traces, metrics, and Cloud Logging, and generates structured incident briefings with cited findings and confidence scores.
When Dynatrace detects an anomaly:
- Collects: Problem metadata, events, and metric timeseries from Dynatrace; correlated log entries from Cloud Logging via trace IDs and timestamps
- Analyzes: Gemini 2.5 Flash on Vertex AI processes pre-summarized evidence through a structured prompt; the model only sees fetched data
- Verifies: A grounding engine checks that every
evidence_idin the output exists in the fetched pool; hallucinated citations are dropped, confidence downgraded - Presents: React UI shows severity badges, confidence levels (high/medium/low), and risk-ranked actions (safe/caution/dangerous); click any claim to open raw evidence in a modal
Result: Mean Time To Investigate drops from 45 minutes to under 30 seconds. Every claim links to original evidence. No black box. No hallucination.
How we built it
Backend: FastAPI with async HTTPX for Dynatrace API calls. The DynatraceClient fetches problems, events, and evidence details via the Dynatrace v2 API. CloudLoggingClient uses the Google Cloud Logging v2 SDK with structured filters for trace correlation.
The IncidentInvestigator class handles the Gemini pipeline. We learned early that dumping 200+ raw evidence items into the prompt produced hard to understand output. The fix: _summarize_evidence_for_prompt() pre-structures evidence by source type with human-readable summaries, reducing tokens and improving reasoning.
The GroundingEngine is the trust layer. It maintains a set of valid evidence_ids and verifies every citation in Gemini's output. Invalid citation? Finding is discarded. For few survivors, confidence is downgraded to "medium" or "low."
Frontend: React, Vite and Tailwind CSS. Incident list with real-time severity badges. Detail view renders the briefing with color-coded confidence badges and risk-ranked action cards. Evidence modal shows raw JSON, proving the analysis is grounded.
Deployment: Single-container Cloud Run service. Dockerfile multi-stages, Node for frontend build, then Python for backend, serving static files via FastAPI's StaticFiles mount. One command: make deploy-quick.
Challenges we ran into
The most difficult challenge wasn't integrating Dynatrace or calling Gemini. It was making the LLM tell the truth.
Attempt 1: Direct prompt with raw JSON evidence. Gemini returned generic restatements of the problem title, citing the problem ID as "evidence" without analysis. Confidence was always "high".
Attempt 2: Strict JSON schema with response_mime_type="application/json". Better structure, but malformed JSON, trailing commas, unescaped quotes, missing closing braces. I added _extract_json() with regex repair and three-attempt retry logic.
Attempt 3: Pre-summarized evidence + explicit "good vs bad finding" examples in the prompt. Produced specific, technical claims. But citations still drifted — IDs mutated in the output.
Final solution: The GroundingEngine verification layer. Post-process every finding. If invalid citation, then Drop it. Surviving findings downgraded if many were dropped. The UI shows exactly which claims passed verification.
Lesson: LLM output is a hypothesis, not a conclusion. Verification is the product.
Accomplishments that we're proud of
Real end-to-end integration: Live Dynatrace synthetic monitor alerts -> Cloud Run deployment -> Cloud Logging correlation -> Gemini analysis -> grounded briefing with verified citations. Not demo data. Not localhost. Production-grade pipeline.
Anti-hallucination architecture: Three-layer defense (input grounding, structured output, citation verification) that drops invalid claims instead of presenting them. This is what makes the system trustworthy enough for actual incident response.
Confidence as a feature: High/medium/low badges are not decorative they reflect evidence strength. One finding with single-source citation is "medium." Same claim with Dynatrace + Cloud Logging corroboration is "high." Engineers now know what to trust.
Risk-ranked actions: context-aware recommendations based on a specific incident's evidence.
What we learned
Evidence summarization beats prompt engineering. Pre-structuring evidence by source type with human-readable summaries reduced Gemini's error rate more than any prompt tweak. The model reasons better when it reads like an SRE notebook, not a JSON dump.
Verification must be automatic, not optional. Manual review of LLM output doesn't scale at 2 AM. The GroundingEngine runs in milliseconds and drops hallucinated findings before they reach the UI. Trust is a feature, not a post-hoc audit.
Structured output is fragile. response_mime_type="application/json" helps but doesn't guarantee valid JSON. Retry logic with temperature escalation, regex repair, and graceful fallback to rule-based analysis are all necessary for production reliability.
What's next for Oncall Agent
Auto-remediation: One-click execution of "safe" actions via Cloud Run API (restart service, scale replicas) with pre- and post-validation
Historical correlation: "This pattern matches 3 incidents last month, all caused by database connection pool saturation. Suggest permanent fix: enable pgbouncer."
Slack integration: Post briefings to
#incidentswith thread-based evidence drill-down; @mention on-call engineer only for "low confidence" findings requiring human judgmentMulti-cloud adapters: Extend to Datadog, New Relic, AWS CloudWatch via pluggable
IntegrationClientinterfaceRunbook generation: Auto-create post-incident runbooks from successful remediation actions, validated against evidence
Log in or sign up for Devpost to join the conversation.