Sonjomon

An autonomous incident-response agent for Google Cloud that knows when to stop.

Comment

Inspiration

At 3am an alert fires. An engineer wakes up, reads several hundred log lines, correlates them against recent deploys, and rolls something back. Most of that work is mechanical — the same handful of failures recur.

But "automate it with an LLM" does not dissolve the problem, it relocates it. The real question is: how much would you let an agent change in production without asking you first?

Give it too little and it is a chatbot that writes summaries. Give it too much and one confidently wrong diagnosis takes down your service at 3am with nobody watching.

I named it Sonjomon — Bengali for restraint.

What it does

The trigger is fully autonomous. A Cloud Monitoring alert publishes to Pub/Sub, which pushes to the agent on Cloud Run. There is no chat interface. No human types anything.

It gathers evidence, then reasons over it. The agent pulls the recent log window from Cloud Logging, four metric series from Cloud Monitoring, and the deployment history from Cloud Run, and hands the lot to Gemini 3.5 Flash in one long-context call. It returns a root cause, a confidence score, cited evidence, and a proposed action.

An autonomy ladder decides what happens next. How far the agent may act alone is a function of two things — how confident it is, and how much damage the proposed action does if that confidence is wrong:

tier = f(confidence, blast_radius)

OBSERVE record findings, take no action SUGGEST recommend to a human, do not execute APPROVE stage the action, execute on explicit approval ACT execute now, then verify independently

Six conditions can only ever push the tier down, never up: the blast-radius ceiling, thin evidence, a similar action that just failed, a third attempt at the same fix, a stale incident, and a global dry-run switch.

Three things the model is not allowed to do:

1.Set its own blast radius. The model proposes an action name; a registry in code supplies the risk. An action not in the registry is rejected outright. 2.Decide whether it may execute. That is deterministic, unit-tested code installed as ADK's before_tool_callback. 3.Grade its own work. Verification is an independent HTTP probe, recorded as a separate event.

Destructive actions are never registered as tools at all. The model cannot call what it was never given.

How we built it

Stack: Gemini 3.5 Flash · Google ADK 2.6.3 · Cloud Run · Pub/Sub · Firestore · Cloud Logging · Cloud Monitoring · Secret Manager · Artifact Registry · Cloud Build

ADK is used as an enforcement seam, not a wrapper:

ADK feature Ladder tier before_tool_callback returns a dict OBSERVE, SUGGEST — blocked with a written reason FunctionTool(require_confirmation=…) APPROVE — ADK pauses for a human callback returns None ACT — proceeds

require_confirmation accepts a callable, so each tool asks the ladder whether this particular call needs a human. Confirmation is a live policy decision, not a static flag.

Decoupling. The push endpoint acks immediately and investigates in a background task. Pub/Sub redelivers on any non-2xx and an investigation outlives the ack deadline, so a synchronous handler would guarantee duplicate agents racing on the same outage.

State. Every incident is an append-only event log in Firestore. Nothing is mutated — a revised diagnosis is a new event, not an edit. The whole incident replays afterwards to answer "why did it do that".

Credentials. The Gemini key lives in Secret Manager, fetched once per process and cached. An env var would expose it to anyone with view access — and this agent holds rollback rights.

Least privilege. The agent's service account can read logs and metrics, call Gemini, write Firestore, and administer one Cloud Run service. It cannot create services, touch IAM, or delete anything.

Data sources: Cloud Logging (no severity floor — the line that explains an incident is often an INFO line just before the errors start), Cloud Monitoring, Cloud Run revision history, and Firestore. The target service never tells the agent what is wrong; chaos markers are filtered out of the evidence it sees.

Challenges we ran into

Cloud Run cuts CPU the moment a request is acked. The background investigation crawled and the instance froze before finishing. --no-cpu-throttling is not optional for a long-running agent. This cost an evening.

Vertex only publishes Gemini 2.5 in the regions available to this project. The hackathon requires 3.5 or newer, so the diagnosis call goes through the Gemini Developer API instead, with the key in Secret Manager.

Least privilege is discovered by breaking things. The agent had run.admin on one service and still could not update it — Cloud Run needs artifactregistry.repositories.downloadArtifacts to pull the image for a new revision. I only found that because the agent tried, failed, and logged the exact permission.

My first chaos injection destroyed its own evidence. The memory-leak fault climbed until the container hit OOM and restarted, which wiped the in-memory fault state and dropped memory to zero before the alert's alignment window closed. The agent kept arriving at a healthy service and correctly reporting "transient, already resolved." The fix was to cap the leak just below the container limit.

Accomplishments that we're proud of

Fourteen live incidents. Fourteen decisions an on-call engineer would have endorsed. Zero wrong actions taken against production.

What it found Tier reached Why that was right v1.5.0 shipped a null-pricing regression; cited the stack trace and counted 262 matching tracebacks SUGGEST → escalated proposed rollback_revision at 95%; high blast radius capped it at APPROVE compared image digests across revisions, worked out which image introduced the typo, and identified the last good revision APPROVE it found the rollback target itself — nobody told it to look at digests three revisions shared the same image digest, so the "healthy" one ran identical broken code ACT — executed escalate_to_human a rollback would have changed nothing; this inference was in neither its registry nor its prompt memory at 97.6%, error rate 702 ACT — ran raise_memory_limit, which failed it saw the failure, refused to claim success, re-diagnosed, and the loop breaker required a human all revisions had traffic_percent: 0 APPROVE a genuine infrastructure misconfiguration, found without being told to look upstream ReadTimeout in 14 of 74 log lines ACT — executed escalate_to_human an upstream fault has no safe fix from this side half the logs None, half 200, error rate spiking then clearing OBSERVE 40% confidence — too low to even recommend, so it said nothing

The friction it removes is the diagnosis, not the fix. An engineer woken at 3am reaches a root cause in twenty to thirty minutes. Sonjomon did it in 8 to 30 seconds, every time, with the file, the line number, the occurrence count, and the offending revision.

What we learned

Long-context reasoning beat pattern matching. The agent's best work was not recognising a known error — it was correlating image digests across five revisions to prove a rollback would achieve nothing.

Confidence thresholds must be calibrated against the real model. My initial ACT floor of 0.90 left that tier effectively unreachable, because Gemini's self-reported confidence clusters in 0.85–0.95 for a well-evidenced diagnosis. Seven live incidents told me what a synthetic test never would.

The hardest thing to demonstrate is an agent correctly doing nothing. Two of the five failure modes have no safe automatic fix by design. Showing an agent succeed is easy; showing one refuse, explain why, and escalate is the harder and rarer claim — and it is the one that decides whether anyone would run this in production.

Why the target is a purpose-built service: orders-api is deliberately fragile, with five injectable failure modes. That was a choice, not a shortcut — it lets the same failure be reproduced on demand and the agent's judgement checked against a known answer. It is also why the agent could find real problems I had not planted, including a traffic misconfiguration and a config mismatch I created by accident while setting up.

What's next for Sonjomon

Point it at a real production service. The next step is attaching it to my own deployed apps and letting it observe genuine failures rather than injected ones.

Learn from approvals. Every APPROVE-tier decision is a labelled example. If a human consistently approves the same recommendation, that is evidence the ceiling for that action could be raised — and evidence the ladder should record rather than assume.

Cost-aware evidence gathering. Currently every alert triggers a full long-context call. A cheaper model could pre-filter and only escalate to Gemini 3.5 when the signals are ambiguous.

Multi-service correlation. Right now the agent reasons about one service at a time. Real outages cascade.

Built solo, in ten days, as a first-year CS student.

Built With

Share this project:

Updates

Submission history