Inspiration
On-call engineering is still painfully manual. When an incident happens at 3 AM, engineers jump between Kibana, dashboards, logs, runbooks, and Slack threads trying to identify the root cause. Most of the time is spent investigating, not fixing the issue.
We wanted to answer a simple question:
What if an AI agent could perform the entire investigation process automatically, learn from previous incidents, and safely remediate known problems before a human even opens their laptop?
What it does
RunBook is an autonomous on-call agent that investigates, correlates, and remediates production incidents in under 45 seconds.
When an anomaly is detected, RunBook:
- Receives alerts from Elastic ML Anomaly Detection
- Investigates the blast radius using ES|QL across affected services
- Eliminates healthy services using a "Time-to-Innocent" approach
- Searches historical incidents using Incident DNA Fingerprinting
- Uses Gemini 2.5 Pro to reason over evidence and calculate confidence
- Automatically remediates known issues when confidence is high
- Escalates uncertain incidents with a complete Chronicle report when human intervention is needed
The platform also includes:
- Shadow Mode for safe validation before automation
- Incident DNA knowledge base
- AI-generated incident narratives
- Multimodal investigation using screenshots and graphs
- Real-time dashboard for monitoring and analytics
How we built it
RunBook combines Google Cloud's agent ecosystem with Elastic's observability platform.
Core Components
- Google Cloud Agent Builder
- Gemini 2.5 Pro
- Elastic 9.4 (ES|QL, ML, Vector Search, Cases, Workflows)
- MCP Server built with FastAPI
- Next.js 16 dashboard
- Supabase for authentication and data storage
- Cloud Run for deployment
Agent Workflow
- Elastic ML detects an anomaly.
- Elastic Workflows triggers the investigation.
- The MCP server exposes specialized tools for querying infrastructure, retrieving incident history, and executing remediation actions.
- Gemini orchestrates the investigation and generates a confidence score.
- High-confidence incidents are automatically remediated.
- Low-confidence incidents are escalated with a complete investigation report.
We also built dedicated Elastic indices for runbook embeddings, incident DNA vectors, blast-radius snapshots, and Shadow Mode predictions.
Challenges we ran into
- Designing safe auto-remediation without creating risk.
- Preventing false positives from triggering production actions.
- Building a confidence-scoring system that engineers can trust.
- Correlating signals across multiple services using ES|QL.
- Creating a reusable incident fingerprinting system that works across different failure patterns.
- Balancing autonomous behavior with human oversight.
- Making investigation outputs understandable instead of producing black-box AI decisions.
Accomplishments that we're proud of
- Built a fully agentic incident-response workflow rather than another alerting tool.
- Created Incident DNA Fingerprinting, allowing the system to learn from previous incidents.
- Developed Shadow Mode, enabling teams to evaluate automation before going live.
- Implemented Time-to-Innocent analysis to rapidly eliminate unaffected services.
- Integrated multimodal investigation using Gemini's vision capabilities.
- Generated Chronicle reports automatically, reducing post-mortem effort.
- Achieved a workflow that can perform in seconds what typically takes engineers tens of minutes.
What we learned
- Trust is more important than automation speed in incident response.
- Engineers need explainability before they will allow autonomous remediation.
- Historical incidents are an extremely valuable source of operational knowledge.
- Elastic's native vector search and ES|QL make large-scale incident correlation surprisingly powerful.
- Combining observability data with LLM reasoning creates workflows that are difficult to achieve using either technology alone.
- Human-in-the-loop design remains essential for production systems.
What's next for RunBook
Our roadmap includes:
- Support for additional observability platforms beyond Elastic.
- More advanced remediation actions for Kubernetes and cloud infrastructure.
- Cross-workspace incident learning while preserving privacy boundaries.
- Slack, Teams, and PagerDuty native integrations.
- Multi-agent collaboration for large-scale incidents.
- Predictive incident prevention using historical DNA patterns.
- Automated post-incident RCA generation and trend analysis.
- Enterprise deployment options with governance and approval workflows.
The long-term vision is to transform RunBook from an incident response assistant into a complete autonomous reliability engineer that continuously learns, investigates, and improves operational resilience.
Built With
- cases)
- elastic-9.4-(es|ql
- fastapi
- gemini-2.5-pro
- google-cloud-agent-builder
- google-cloud-run
- google-kubernetes-engine-(gke)
- mcp-(model-context-protocol)
- ml-anomaly-detection
- next.js-16
- postgresql
- python
- react-19
- supabase
- tailwind
- typescript
- vector-search
- workflows
Log in or sign up for Devpost to join the conversation.