runbook-executor

Architecture

Inspiration

Every on-call rotation has the same shape: something breaks, a human gets paged at 2am, and the fix is already written down somewhere — a runbook, a wiki page, a pinned Slack message. The engineer isn't diagnosing a mystery, they're just the one manually typing the commands the team already agreed on: rotate the leaked key, kick the dependent service so it picks up the new one, tell everyone what happened.

That gap — between "we already know what to do" and "someone still has to do it by hand" — is exactly the kind of friction this hackathon's Taskmaster track is asking for. We didn't want to build another chatbot that tells an on-call engineer what it would do. We wanted an agent that reads the runbook and actually does it, against real infrastructure, with a paper trail a human can check afterward.

What it does

Runbook Executor is a webhook that turns an incoming alert into a completed remediation.

An alert — type, severity, resource, plus context like a secret_id or a list of dependent_services — hits POST /alert. The agent looks up the matching runbook by trigger_type, from Firestore (or the local YAML files before Firestore's been seeded). Gemini 3.5 Flash reasons over the runbook's ordered steps and decides which one applies to this alert. It calls the matching tool against real Google Cloud infrastructure — rotate_secret against Secret Manager, or redeploy_service against Cloud Run — no simulation layer, the whole point is that it actually happens. Every action gets written to Cloud Logging with the reasoning attached, and optionally posted to a chat webhook, before the agent moves to the next step. If a step is unconfirmed, fails, or no runbook matches the alert at all, the agent stops and escalates to a human instead of improvising. It follows the documented procedure or it hands off — it never freelances.

Two demo scenarios ship with it: a leaked credential (rotate the secret, redeploy the service that depends on it) and a service outage (escalate). Both are one command away with scripts/fire_test_alert.py.

How we built it

The core is a Google ADK Agent (runbook_executor/agent.py) wired to five tools: get_runbook, rotate_secret, redeploy_service, log_and_notify, and escalate_to_human. A thin FastAPI app (server.py) wraps it with InMemoryRunner so an alert becomes a single structured message into the agent and its tool calls become the response transcript.

The tools talk to real GCP client libraries directly: google-cloud-secret-manager for rotation, google-cloud-run for forcing a fresh revision (Cloud Run only re-resolves a :latest secret reference on a new revision, so the redeploy step nudges a label to trigger one), google-cloud-firestore for the runbook lookup with a local-YAML fallback so the project is runnable before Firestore is seeded, and google-cloud-logging for the audit trail, with a stdout fallback so a demo never dies on a logging hiccup.

The agent's instruction prompt is deliberately a policy document, not a personality: match the alert to a runbook step, call the one real tool that step names, log why, and stop and escalate the moment anything is unconfirmed. Deploy target is Cloud Run via a Dockerfile and deploy/deploy.sh.

Challenges we ran into

Getting the agent to actually act rather than describe took the most iteration. It's easy to build something that reasons well about a runbook and then narrates a plan in prose — the harder, more valuable thing is an agent whose tool calls are the only way it's allowed to report success, so "the agent said it rotated the secret" and "the secret was rotated" can't drift apart.

The second challenge was scope control on the policy prompt itself. An agent that's allowed to improvise past what the runbook says is a liability the moment it's touching Secret Manager and Cloud Run for real — so the instruction explicitly forces escalation on any unmatched runbook or unconfirmed step, rather than letting Gemini try to be helpful and guess.

google-adk's API is also moving fast — we had to keep checking the installed package's own docs against what the public examples showed, since the Agent / Runner shapes shift release to release.

Accomplishments that we're proud of

The thing we're most proud of is the "execution path" in the architecture diagram — the agent doesn't stop at a recommendation, it calls Secret Manager directly and only writes the audit entry after the fact. That's the difference between a demo that talks and a demo that works. We're also glad the escalation path isn't an afterthought: a runbook-executor that can't say "I don't know, paging a human" is more dangerous than no automation at all.

What we learned

We came in thinking the hard part would be prompting Gemini to pick the right step. It wasn't — the hard part was designing tool contracts specific and constrained enough that the agent literally cannot succeed without the real side effect happening, and an audit trail that makes the agent's reasoning inspectable after the fact rather than a black box. We also learned a lot about Cloud Run's revision model along the way — specifically that a secret reference pinned to :latest doesn't just pick up a new version on its own; something has to trigger a fresh revision.

What's next for runbook-executor Real alert sources — a PagerDuty or Sentry webhook instead of the simulated fire_test_alert.py trigger. More runbook actions beyond rotate + redeploy: scaling a service, rolling back a bad revision, IAM policy fixes. A real credential-issuing call in rotate_secret instead of the placeholder timestamp value, for teams that want end-to-end-real rotation. A proper Slack/Chat app instead of a plain incoming webhook, so a human can approve a risky step inline rather than only being notified after the fact. Multi-project support, so one deployment can be the on-call agent for a whole fleet of services instead of a single GCP project.