Elastic On-Call Agent: Agentic Ops with Google Cloud

A Gemini-backed incident triage agent that uses Elastic evidence, Cloud Run, and Slack to move from alert to root cause to safe next action.

Comment

Inspiration

Production incidents rarely fail because teams cannot see an alert. They fail because the first triage loop is slow.

An on-call engineer receives an alert, then manually checks logs, metrics, recent deployments, runbooks, service health, customer impact, and similar past incidents. Those signals are usually scattered across different tools. During the first critical minutes, the engineer needs to answer four questions quickly:

What changed? What broke? Which evidence supports the probable root cause? What is the safest next action?

Elastic On-Call Agent was inspired by this operational problem. The goal is to turn Elastic into an operational evidence layer, use Gemini on Google Cloud as the reasoning layer, and run a controlled remediation workflow on Cloud Run after human approval.

What it does

Elastic On-Call Agent is a Gemini-backed incident triage and remediation assistant for on-call teams.

The hosted demo runs a real demo workload on Google Cloud Run:

payment-api checkout-api agent-ui

The demo starts with a healthy checkout flow. The agent-ui performs live health checks against the real Cloud Run services and verifies that:

checkout-api is healthy payment-api is healthy customer checkout flow is successful

When the user clicks Simulate incident, the agent triggers a controlled failure mode in the real checkout-api service. The checkout-api moves into a redis_timeout failure state. As a result:

checkout-api becomes critical customer checkout flow fails payment-api remains healthy

The agent then combines live workload health with operational evidence from Elastic. Gemini 2.5 Pro on Vertex AI generates an incident analysis that includes:

  • incident summary
  • probable root cause
  • affected services
  • evidence
  • timeline
  • confidence level
  • safe next action
  • human approval requirement

The user can ask follow-up questions in the Ask Agent panel. The answer is generated by Gemini and grounded in the available incident evidence.

When the user clicks Apply remediation, the agent performs a controlled remediation action against the demo checkout-api, verifies the workload again, confirms the customer checkout flow is healthy, and creates a GitHub issue for engineering follow-up.

The remediation is intentionally human-approved. The agent does not silently perform production-changing actions.

How we built it

The project is built with Python and FastAPI and deployed on Google Cloud Run.

The system is composed of three Cloud Run services:

payment-api: a demo payment authorization service checkout-api: a demo checkout service that depends on payment-api agent-ui: the incident triage, reasoning, remediation, and demo control plane

Elastic Cloud Serverless is used as the operational evidence layer. The agent collects and structures incident evidence such as logs, alerts, deployment context, runbooks, and related incident signals.

Gemini 2.5 Pro on Vertex AI is used as the reasoning layer. The backend creates a structured evidence bundle and sends it to Gemini. Gemini returns an evidence-backed incident explanation, probable root cause, hypotheses, and safe next actions.

Google Cloud Run is used as the execution layer. The demo workload and the agent control plane are all deployed as managed Cloud Run services.

Google Secret Manager stores sensitive configuration such as Elastic API keys, Slack webhook values, and GitHub tokens. No real secret values are committed to the repository.

GitHub integration is used for engineering follow-up. After remediation is applied and verified, the agent creates a GitHub issue containing incident context and remediation results.

The public demo also includes basic safeguards:

  • human approval before remediation
  • demo-only controlled remediation endpoint
  • Secret Manager-based secret handling
  • no customer data
  • no internal data
  • no destructive production action

Architecture:

Cloud Run agent-ui -> live workload health checks -> checkout-api on Cloud Run -> payment-api on Cloud Run -> Elastic evidence collection -> Gemini 2.5 Pro on Vertex AI reasoning -> human-approved remediation -> post-remediation verification -> GitHub issue creation

Detailed flow:

Health check -> checkout-api /livez -> payment-api /livez -> checkout-api /checkout

Simulate incident -> checkout-api /admin/fail -> checkout-api enters redis_timeout mode -> customer checkout flow fails

Ask Agent -> collect incident evidence -> send structured evidence to Gemini -> return grounded incident analysis

Apply remediation -> checkout-api /admin/repair -> verify checkout-api /livez -> verify payment-api /livez -> verify checkout-api /checkout -> create GitHub issue

Challenges we ran into

The biggest challenge was making the demo realistic without using customer data, internal data, or confidential production data.

To solve that, the project uses a real but isolated demo workload. The payment-api and checkout-api are real Cloud Run services, but the data and incident scenario are synthetic and safe for a public hackathon demo.

Another challenge was moving from a static incident simulation to a real operational flow. The demo now has a real service dependency, a real failure mode, a real remediation endpoint, and real post-remediation verification.

A third challenge was making the agent safe. Automatic remediation can be risky, so the project keeps remediation behind an explicit human approval step. The agent can reason, recommend, execute a controlled demo repair, and verify the result, but the user must approve the action.

We also ran into Cloud Run-specific deployment and runtime issues, including service-to-service URL configuration, Secret Manager permissions, and health check path behavior. We solved this by using explicit Cloud Run service URLs, Secret Manager IAM bindings, and /livez endpoints for workload validation.

Accomplishments that we're proud of

We built a complete end-to-end incident triage and remediation workflow:

real Cloud Run demo workload payment-api and checkout-api service dependency live workload health checks controlled incident injection Elastic evidence collection Gemini 2.5 Pro reasoning Ask Agent follow-up Q&A human-approved remediation post-remediation verification GitHub issue creation public Cloud Run demo public GitHub repository

The final system is not just an alert formatter. It reasons over operational evidence, identifies the likely failing service, recommends a safe action, executes a controlled remediation after approval, verifies the customer flow, and creates an engineering follow-up issue.

The demo also clearly separates responsibilities:

Elastic is the evidence layer. Gemini is the reasoning layer. Cloud Run is the workload and execution layer. GitHub is the engineering follow-up layer. Human approval is the safety layer.

What we learned

The most important design lesson was that incident response agents need grounding.

A language model alone should not guess the root cause of an incident. It needs structured operational evidence and live workload state. Elastic is strong at storing and retrieving operational evidence. Gemini is strong at reasoning over that evidence and explaining it in a way engineers can act on.

We also learned that agentic operations should not mean uncontrolled automation. The safest pattern is:

observe collect evidence reason recommend ask for approval execute a controlled action verify the result create follow-up

This pattern keeps the agent useful without bypassing operational safety.

What's next for Elastic On-Call Agent: Agentic Ops with Google Cloud

Next improvements would include:

real Elastic alert webhook integration Elastic MCP Server integration for tool-based evidence retrieval GitHub deployment history integration richer GitHub issue templates Jira integration postmortem draft generation human approval workflow for rollback preparation multi-service blast radius analysis more incident scenarios beyond Redis timeout regression support for multiple environments and teams service-to-service authentication between Cloud Run services Cloud Trace and Cloud Logging correlation stronger rate limiting and abuse protection for public demos

Built With

Share this project:

Updates