Inspiration
Modern incident response is still surprisingly manual. An alert fires, an engineer opens logs, checks recent deployments, compares metrics, searches through code changes, decides whether to restart or roll back a service, verifies recovery, and finally writes an incident report.
SentinelOps was created to automate that workflow.
Instead of building another chatbot that only suggests what an engineer should do, SentinelOps acts as an autonomous AI incident commander. It detects an operational problem, gathers technical context, investigates the likely root cause, selects a safe remediation strategy, executes approved actions, verifies that the system recovered, and records what happened.
What it does
SentinelOps provides an event-driven incident response workflow:
- Receives an incident or monitoring event.
- Collects logs, metrics, deployment information, and recent code changes.
- Uses Gemini to analyze the evidence and determine the likely root cause.
- Coordinates specialized agents for infrastructure, logs, and source-code analysis.
- Creates a remediation plan based on impact and confidence.
- Executes low-risk recovery actions automatically.
- Requires human approval for potentially destructive operations.
- Verifies service health after remediation.
- Stores incident state and execution history.
- Generates a structured post-incident report.
A typical demonstration scenario is a faulty application deployment that causes elevated HTTP 500 errors. SentinelOps identifies the failing revision, correlates the incident with the latest deployment, recommends or performs a rollback, validates that the service has recovered, and produces an incident timeline.
How we built it
SentinelOps is designed as a multi-agent system using Google's Agent Development Kit and Gemini.
The coordinator agent is responsible for the overall incident lifecycle and delegates tasks to specialized agents.
Incident Coordinator
Controls the workflow:
Detect → Investigate → Decide → Remediate → Verify → Report
Log Analysis Agent
Examines Cloud Logging data and extracts relevant errors, exceptions, latency anomalies, and failure patterns.
Infrastructure Agent
Inspects the state of the deployed service and determines whether infrastructure-level remediation is appropriate.
Code Analysis Agent
Correlates incidents with recent source-code and deployment changes to help identify regressions.
Remediation Agent
Evaluates available recovery actions such as restarting a workload, rolling back a deployment, or preparing a code fix.
Verification Agent
Runs health checks after remediation and determines whether the incident is actually resolved.
Gemini is used for reasoning, evidence correlation, root-cause analysis, and remediation planning.
Google Cloud provides the execution environment and operational services required by the agent workflow.
Safety by design
Autonomous infrastructure operations require guardrails.
SentinelOps separates actions by risk level.
Low-risk actions can be executed automatically, while high-impact actions require explicit human approval.
Examples:
- Read logs → automatic
- Analyze metrics → automatic
- Run health checks → automatic
- Restart a failed test service → automatic
- Create a remediation recommendation → automatic
- Modify production data → approval required
- Destructive infrastructure changes → blocked or approval required
Every action is recorded in the incident timeline so the system remains observable and auditable.
Challenges we ran into
One of the main challenges was designing an agent that does more than generate recommendations.
Incident response requires a closed feedback loop. Executing a remediation action is not enough — the system must observe the result and determine whether the service actually recovered.
Another challenge was balancing autonomy with operational safety. Giving an AI agent unrestricted infrastructure access would be dangerous, so SentinelOps uses explicit tool permissions and risk-aware approval boundaries.
We also had to keep the workflow reproducible so that an incident can be demonstrated consistently during judging.
What we learned
Building SentinelOps reinforced that useful operational agents need more than an LLM.
They need:
- reliable tools
- structured state
- event-driven execution
- explicit permissions
- observable actions
- verification loops
- persistent incident history
The most important design principle became simple:
An incident is not resolved when the agent proposes a solution. It is resolved when the system proves that the service is healthy again.
What's next for SentinelOps
The next steps include:
- deeper Cloud Monitoring integration
- GitHub pull-request remediation
- Kubernetes and GKE support
- automated rollback policies
- historical incident memory
- similarity search across previous incidents
- Slack and PagerDuty integrations
- policy-based production approvals
- OpenTelemetry tracing
- automatic postmortem generation
The longer-term goal is to make SentinelOps a reliable autonomous operations layer that works alongside DevOps and SRE teams rather than simply providing another monitoring dashboard.
Log in or sign up for Devpost to join the conversation.