KaiOPS — Multi-Cloud SRE Agent Fleet

Inspiration

I've been the person paged at 3 AM about a cluster I didn't build, at a company with no SRE team, with nobody to escalate to. You open six tabs — two cloud consoles, Grafana, ArgoCD, GitHub, Kubernetes — and correlate timestamps by hand at the hour when humans are worst at it.

Most of that work is mechanical. Pattern matching across four data sources. We now have systems that are genuinely good at that. But nobody wants to give a language model unsupervised write access to production — and they shouldn't.

KaiOps exists because I wanted the teammate I didn't have: one that watches, investigates, explains, and then asks before it acts.

What it does

When a deployment goes unhealthy — ArgoCD Degraded, CrashLoopBackOff, OOMKilled — KaiOps detects it automatically and runs a full root cause analysis across GCP, AWS and Azure without a human starting it.

The loop: webhook fires → job lands in Firestore (PENDING) → a worker claims it via compare-and-swap (no double-claim) → the orchestrator identifies the app's cloud provider and delegates to the right specialist → that specialist pulls logs, checks live Grafana alert state, queries Prometheus metrics, and grounds the diagnosis against a runbook corpus in Gemini Enterpirses Agent Platform → an RCA report posts to Slack with a console deep-link → a human approves or rejects the suggested fix → only then does remediation execute through GKE, EKS or AKS native APIs.

No human is in the loop until the approval gate. The trigger, the investigation, the cross-cloud evidence gathering and the report all happen unattended.

Governance (Fortified Enterprise Fleet): Agent Registry (platform + custom, 43 TOOL_SPEC tools), Agent Identity (SPIFFE principal), Semantic Governance (NL constraint bound to agent identity), Model Armor (template provisioned, app-layer enforcement — honestly disclosed), Agent Gateway (egress allowlist + IAP), and per-app Grafana dashboards + Prometheus alerts auto-provisioned on registration.

How we built it

Model & framework: Gemini 3.6 Flash via Gemini Enterprise Agent Platform , orchestrated with Google ADK (LlmAgent, sub_agents, FunctionTool with require_confirmation=True for the HITL gate).

Agent design: A root orchestrator delegates deep investigation to three cloud-specific specialist sub-agents (GCP, AWS, Azure) via A2A. Shallow lookups use deterministic Python routers — the app's cloud provider is a database field, not a model decision. Two paths on purpose: save inference for decisions that genuinely need judgment.

Tool isolation: 43 tools behind 6 IAM-private Cloud Run MCP servers (ArgoCD, AWS, Azure, GCP, GitHub, Grafana). Each is independently deployed and separately authorised. The blast radius of a misbehaving agent is bounded by IAM, not by a prompt.

Infrastructure: Cloud Run (backend, frontend, 3 A2A specialists, 6 MCP servers), GKE + CI/CD, Firestore (metadata, ADK sessions, job state machine), Gemini Enterprise Agent Platform Agent Engine, Gemini Enterprise Agent Platform (RAG), Secret Manager, Cloud Build, Artifact Registry, IAP, Cloud Logging & Monitoring.

Failure tolerance: Status-guarded compare-and-swap on job claim (no double-claim), bounded job timeout (900s), max-attempt cap (2), terminal FAILED state with error recorded.

Built solo over the submission period.

Challenges we ran into

The Agent Gateway wall. I spent close to two weeks trying to route agent-to-agent traffic through the managed Agent Gateway with per-agent identity. Every call failed with a generic egress denial. The gateway decision log revealed clientCertChainVerified: false with a certificate validity window starting at the Unix epoch (1970) — rejected upstream at Google's managed Secure Web Proxy, not in my configuration. I probed all four agent_spec.type values; A2A_AGENT exists but is not exposable through the services registry.

One agent with 40 tools. My first architecture put every tool on a single agent. It looked like intelligence — until it picked the AWS log reader for an Azure app, got an empty result, and confidently reported "no errors found." Worse than an error, because an error you can see. The fix: three specialists, each owning only its own provider's tooling, with separation enforced by what exists rather than by what the prompt asks.

Concurrency. Two workers polling the same job queue can both start the same RCA and double-execute a remediation. Fixed with a status-guarded compare-and-swap on job claim, plus bounded timeouts and max-attempt caps — old patterns from distributed systems, applied to an agent runtime.

Accomplishments that we're proud of

• The autonomous loop is real. Not a chatbot. A pod dies, nobody asks anything, and a Slack report lands with a correct diagnosis, live Grafana metrics, and a suggested fix waiting for approval. That's the teammate I wanted.
• Cross-cloud routing that actually routes. The ArgoCD poller infers AKS/EKS/GKE from the cluster destination and routes both the RCA and the remediation to the right cloud with no manual metadata edits. Most agents in this space are single-cloud.
• Governed destructive actions. restart_pod, rollback_application and sync_application sit behind ADK's native require_confirmation gate with single-use tokens, session/user binding, and cross-user tamper resistance. A model cannot mutate production without a human signing off.
• The gateway investigation. Finding clientCertChainVerified: false in a decision log, tracing it to a 1970-epoch certificate rejected at the managed SWP layer, and probing all four agent_spec.type values to characterize the limitation — that's the kind of debugging I'm most proud of, and it's fully documented.
• Honest documentation. Every platform limitation — gateway, Model Armor binding, Memory Bank scope, shared A2A credential — is disclosed in the README and docs/ with evidence. Not a single overclaim.

What we learned

• The interesting question in agent design isn't "how much can the model decide" — it's the smallest set of decisions that genuinely require judgment. Everything else should be code, because code is testable.
• Prompts are not a security boundary. They're a suggestion with good intentions. Put destructive actions behind IAM, behind a confirmation gate, behind something that can't be argued with.
• Read the logs, not the docs. The docs told me what should work; the gateway decision log told me what did.
• Write down what didn't work. It's the most credible thing in your repository.

What's next for KaiOPS — Governed Multi-Cloud SRE Agent Fleet

• Full identity-routed A2A mesh. Once A2A_AGENT is exposable through the platform registry, replace the shared-credential mesh with per-agent identity on every hop — the architecture is already designed for it.
• Cross-session recall. Wire managed Memory Bank into the Cloud Run demo path so incident N+1 can recall what was learned from incident N — the ADK tools are ready, the integration is env-gated.
• Predictive mode. Today KaiOps reacts to failures. The Grafana and Prometheus data is already flowing — the next step is anomaly detection that triggers an investigation before the pod crashes.
• More clouds, more tools. The MCP mesh is designed to grow. Oracle Cloud, Datadog, PagerDuty, Jira — each is a new Cloud Run service with its own IAM boundary, not a function bolted onto the root agent.
• Open source the governance patterns. The HITL gate, the CAS job queue, the deterministic routing layer — these aren't KaiOps-specific. They're patterns any agent builder should be able to reuse.

Built With

• a2a
• artifact-registry
• cloud-build
• cloud-logging
• cloud-monitoring
• cloud-run
• fastapi
• firestore
• gemini-3.6-flash
• geminienterpiseagentplatform
• gke
• google-adk
• iap
• mcp
• python
• react
• secret-manager
• typescript
• vertex-ai
• vertex-ai-agent-engine
• vertex-ai-search
• vite