Inspiration
Production incidents are rarely difficult because of a single alert. The real challenge begins after detection: collecting evidence, understanding what changed, deciding what action is safe, coordinating approval, executing remediation, and proving that the service actually recovered.
Most incident-response tools stop somewhere in the middle. They alert, open a ticket, suggest a possible cause, or provide a chatbot for an engineer to consult.
We wanted to build something that closes the entire loop.
That became SentinelOps — an autonomous AI incident commander designed around one core principle:
An incident is not resolved when the agent proposes a solution. It is resolved when the system proves that the service is healthy again.
What it does
SentinelOps manages the complete incident lifecycle:
Detect → Investigate → Decide → Remediate → Verify → Report
In the live workflow, a broken Cloud Run revision begins returning HTTP 500 responses. SentinelOps then:
- detects the failure through Cloud Logging;
- receives the event through Pub/Sub;
- correlates duplicate signals into a single incident;
- uses Gemini and Google ADK to investigate the issue;
- gathers evidence from logs, infrastructure, deployment context, and service state;
- produces a structured root-cause hypothesis and remediation plan;
- applies safety policies to determine whether approval is required;
- pauses at a human approval gate for high-impact actions;
- performs an allowlisted rollback to a known healthy Cloud Run revision;
- runs a real health check after remediation;
- marks the incident resolved only when verification succeeds;
- generates a final incident report.
The dashboard exposes the full lifecycle so an operator can see what happened, what the agents concluded, what action was proposed, what was approved, and whether the service actually recovered.
How we built it
SentinelOps is built around a separation between reasoning, control, and execution.
The agent layer uses Google Agent Development Kit (ADK) with Gemini. Instead of relying on a single general-purpose agent, the system uses an Incident Commander together with specialized agents focused on:
- log analysis;
- infrastructure context;
- deployment and code context;
- remediation planning;
- verification planning.
The control plane runs on Google Cloud Run and uses:
- Cloud Logging / Log Router for failure detection;
- Pub/Sub for event delivery and workflow events;
- Firestore for persistent incident state;
- Vertex AI / Gemini for investigation and reasoning;
- Cloud Run Admin API for constrained remediation;
- FastAPI for the backend and dashboard API.
The workflow is event-driven and stateful. Incident state persists across the full lifecycle instead of treating every AI call as an isolated interaction.
Challenges we ran into
One of the biggest challenges was making the system genuinely end-to-end rather than building a simulated AI demo.
The complete flow had to work for real:
HTTP 500 → Cloud Logging → Pub/Sub → AI investigation → approval → rollback → HTTP 200
That introduced several engineering problems at once:
- asynchronous event delivery;
- duplicate detector events;
- persistent workflow state;
- Cloud IAM permissions;
- Cloud Run revision metadata;
- safe execution boundaries;
- structured model output;
- post-remediation verification.
Another challenge was working with tool-using agents and structured responses. An agent naturally produces tool calls and intermediate results, while the application needs a predictable structured incident analysis.
We solved this by separating investigation from final structured formatting instead of forcing a single agent step to both use tools and satisfy the final schema.
Safety was another major challenge. Giving the model unrestricted cloud access would have made implementation easier, but it would have undermined the entire point of a production-oriented incident commander. We instead built deterministic policy checks, allowlists, explicit approvals, and verification requirements around the AI layer.
Accomplishments that we're proud of
The main accomplishment is that SentinelOps does not stop at diagnosis or recommendation.
It completes a real closed-loop incident response workflow.
We are particularly proud that the system can:
- detect a real Cloud Run failure automatically;
- investigate it using Gemini and specialized ADK agents;
- maintain a persistent incident lifecycle;
- require approval for risky actions;
- execute a real rollback rather than simulate one;
- verify the recovered service with a real health check;
- refuse to report an incident as resolved if verification fails.
We are also proud of the safety model. The model never receives unrestricted infrastructure access. Read-only inspection and high-impact execution are deliberately separated, with deterministic controls around remediation.
The result is not just an AI assistant that talks about incidents, but a system that can safely coordinate and complete incident-response work.
What we learned
The biggest lesson was that autonomous agents are mostly a systems-engineering problem.
The model is only one component.
A useful production agent also needs:
- durable state;
- deterministic transitions;
- idempotent event handling;
- scoped tools;
- permission boundaries;
- human approval where appropriate;
- observability;
- reliable verification.
We also learned that verification should be treated as a first-class workflow stage.
An API returning success does not prove that a service recovered. The service itself must provide evidence of recovery.
That changed the architecture from:
Detect → Recommend
into:
Detect → Investigate → Decide → Remediate → Verify → Report
That final verification stage is what turns an AI recommendation system into an actual incident-response system.
What's next for SentinelOps
The current implementation focuses on Cloud Run, but the architecture is designed to expand beyond a single platform.
Next steps include:
- support for additional cloud and infrastructure providers;
- more observability and telemetry sources;
- additional remediation strategies;
- longer-running incident workflows;
- historical incident memory;
- cross-service incident correlation;
- policy-driven automation levels;
- stronger auditability and agent observability;
- support for larger multi-service environments.
The long-term goal is not to replace operators.
It is to remove repetitive investigation, coordination, and verification work while keeping humans in control of decisions that carry real operational risk.
Built With
- ai-agents
- autonomous-agents
- cloud-infrastructure
- cloud-logging
- cloud-run-admin-api
- devops
- docker
- event-driven-architecture
- fastapi
- firestore
- gemini-3.5-flash
- github-api
- google-adk
- google-cloud-run
- human-in-the-loop
- incident-response
- multi-agent-systems
- observability
- pub/sub
- python
- rest-api
- safety-guardrails
- sre
- vertex-ai
- workflow-automation
Log in or sign up for Devpost to join the conversation.