Warden

• 

  Four agents, thirteen tools, one boundary. Nothing crosses it without a merged pull request.

Inspiration

Most production incidents are not interesting. Somebody is awake at three in the morning reading logs, and forty minutes later they find a typo that shipped that afternoon. The fix takes one line. The finding took forty minutes.

So the obvious idea is to let an AI agent do the reading. The problem is the next sentence. Nobody sane hands production credentials to a language model, and every demo we had seen quietly did exactly that, or avoided the question by never touching anything real.

We wanted to answer the question instead of dodging it. Not "can an agent fix this," but "what is the most you can safely let it do, and how do you prove the limit is real?"

What it does

Warden is a fleet of four agents that watch a live Kubernetes cluster.

Triage goes first and sees nothing but the alert text. It has no access to the cluster at all, so it cannot be talked into acting on something it cannot see. It decides whether this is real.

The diagnostician reads the pods, the logs and the deploy history. Read access only. It cannot write to the cluster and it cannot write to the repository. In our demo run it found a single missing character in a URL, named the exact commit that introduced it, and counted two hundred and forty nine container restarts. Every claim it makes points back to the log line it came from.

The remediator writes the smallest patch that fixes it and opens a pull request. Three files, twelve lines. That size limit is checked against the actual diff, not politely requested in a prompt.

Then the fleet stops. It does not stop because it ran out of steps. It stops because no credential in the system lets it go any further. A human reads the pull request and merges it. Continuous integration applies it using a service account scoped to one namespace with no delete permission, and that account is the only thing in the entire system that can change the cluster. No agent holds it.

Afterwards the verifier checks the cluster itself rather than trusting that the fix worked. If the cluster had not recovered, it would open a revert. Propose one. Never apply one.

How we built it

The whole fleet runs on Google's Agent Development Kit, with Gemini through Vertex AI. The important piece is one function. Every tool call from every agent passes through a single before_tool_callback, and that callback is where policy lives. It checks the agent's manifest, the budget, the patch size and the kill switch before the tool ever runs.

There are thirteen tools. Each agent is granted only the ones its manifest allows, and the remediator gets three. Two tools, delete and scale, are granted to nobody at all.

That was deliberate and it is the part we would defend hardest. If we had simply never written a delete function, then saying the agent cannot delete things would be true and it would prove absolutely nothing. So the capability is real, fully implemented, and granted to no one. During the live run the model hallucinated a delete_workload call anyway, and the policy layer refused it. That refusal is visible in the dashboard. You cannot test a lock by pointing at a wall.

Google Cloud runs the control plane. Cloud Run hosts the service, Firestore holds state, Pub/Sub delivers alerts, Cloud Scheduler runs a periodic sweep, and Secret Manager holds every credential. The Pub/Sub push endpoint verifies the OIDC token of its caller, so the alert ingress authenticates who is talking to it rather than trusting anything that reaches the URL.

The cluster it looks after is Azure Kubernetes Service, on purpose, to make the point that this works on any Kubernetes. Both clouds are built with Terraform, and the repository the agents write to is a real GitOps repository with a protected main branch.

Challenges we ran into

The deployment fought us for most of a day. Cloud Run pins image digests, so pushing a new image tagged latest changed nothing and we spent an embarrassing amount of time looking at old code. The build context was one point one gigabytes because a virtual environment with forty two thousand files was being uploaded every time. Docker preserved host file permissions on copy, so the container started as a non root user and immediately could not read its own policy module. Secret Manager happily created empty secret containers with no versions inside them, and Cloud Run refused to start with an error that did not obviously say so.

The agent side had better failures. On one run the remediator read the same repository file seven times in a row and burned through its token budget. It did not crash. The budget cap caught it, refused the next call and closed the incident cleanly, which is exactly what the cap is for, but it taught us that a budget is a safety mechanism and not an accounting line.

The most useful failure was about identity. We tested whether a personal access token could push straight to the protected main branch. It could. HTTP two zero one, straight through the branch protection, because a token carries the permissions of whoever created it. The GitHub App gets four zero nine on the same operation, because an App has no permissions of its own. That difference is the whole reason the remediator runs as an App and not as a token, and we only know it because we tried it.

Accomplishments that we're proud of

A complete live run, end to end, with nothing staged. We broke a real cluster with one character, the alert fired, four agents diagnosed it, and pull request number ten was opened by warden remediator, a bot account, not by us. Thirteen tool calls, sixty two thousand tokens, roughly half a cent.

The denial that appears in that run is not scripted. The model asked for a tool it was never given, and the system said no, and both halves of that are visible on screen.

One hundred and twenty three tests pass. Google Cloud costs under a dollar a month to run, measured, and the entire agent side of a full incident costs less than a cent.

Mostly though, we are proud that the honest answer to "what if the model goes wrong" is not a promise. It is that nothing in the system holds a credential that would let it matter.

What we learned

That the interesting safety work is not in the prompt. Every constraint we tried to express as instructions to the model was eventually ignored by the model, usually politely. Every constraint we moved into the callback held. The patch size limit is the clearest example. As a sentence in a prompt it was a suggestion. As a check against the real diff it is a fact.

We also learned that a governance story only counts if the capability being refused actually exists, and that the boundary is worth more than the automation. Four agents doing the thinking is useful. Four agents doing the thinking while the credential stays with the human is the thing worth building.

What's next for Warden

Warden only changes deployment configuration today, and never application code. That covers a large share of how real incidents actually begin, but the boundary should be enforced in the tools rather than in what the agents happen to have access to.

Next is a wider set of estates rather than one cluster, evidence collection that survives across incidents so the diagnostician learns the failure patterns of a specific service, and a proper audit export so the whole decision trail leaves the system in a form a compliance team can read.

The scheduled sweep also needs work. Right now it can open an incident but never escalates one, because the message it carries has no symptom in it. It is a real gap and we would rather name it than hide it.

Built With

• fastapi
• javascript
• python
• yaml