Warden - an authorization gateway for agentic fleets

An authorization gateway for a Travel & Expense agent fleet - based on Gemini, Google's ADK, Cloud Run and Firestore. Every tool call an agent makes is intercepted, checked against spend policy.

Comment

Inspiration

2026 didn't wait long to produce real examples of agents doing things nobody had actually authorized. An assistant exploited a gym's booking system to jump a waitlist, unprompted. A separate incident had an OpenAI agent using exposed credentials across four services during a breach. At the same time, corporate agent fleets are being handed real spending authority - booking travel, holding rooms, charging cards - and the solution converges to task-bounded authorization for what an agent can actually spend, and on what.

I wanted to build a small, working version of that idea, scoped to a workflow every finance org already has a strict policy for: Travel & Expense. Not a policy document - a gateway that actually intercepts every tool call before it executes.

What it does

Warden sits in front of a fleet of task agents - search, booking, hotel, cab, and payment agents - and checks every tool call before it reaches the real tool, not after. Each call goes through a deterministic hard-limit check in plain code, then a cheap Gemma pass that clears obviously in-scope calls for free, then a careful Gemini review (via Google's Agent Development Kit) for anything ambiguous, reading the fleet's actual policy document. An allowed call gets a signed, task-bounded token. A blocked call goes to a second ADK agent that decides live — not scripted per route — whether to retry through the correctly-scoped agent or abort the trip.

Every decision lands in a hash-chained Firestore ledger before anything happens next, and a live dashboard shows the whole pipeline running in real time - the call trace, the transaction history, and every real touch of Google Cloud infrastructure behind each decision.

How we built it

  • Five task agents — search, booking, hotel, cab, payment — each with a declared tool scope, built as Google ADK agents.
  • Every tool call routes through a gateway before reaching the real tool: hard-limit check → Gemma triage → Gemini review (ADK reviewer agent, reading the fleet's policy doc) → signed token or block.
  • A blocked call goes to a second ADK agent, the orchestrator, which decides live whether to retry through the correctly-scoped agent or abort.
  • A live dashboard renders the whole pipeline in real time — call trace, transactions, Google Platform activity — driven by one Server-Sent Events (SSE) stream the gateway publishes to at every stage.
  • Firestore backs the agent registry and a hash-chained audit ledger; every decision persists before the next step runs.
  • Deployed on Cloud Run, scaled to zero, running Gemini through Google AI Studio.
  • Vertex AI Model Armor wired in as an opt-in extra screening layer on the reviewer's calls.
  • The same reviewer separately deployed to Vertex AI Agent Engine for a real per-agent cryptographic identity and a persistent Memory Bank — both auto-provisioned by Google, not configured by hand.
  • Local development ran against small open models on a laptop, through Ollama — same code, same agent definitions, only MODEL_BACKEND swapped for what's actually deployed.

Challenges we ran into

Most of the real time went into bugs that only showed up by actually running the thing against live infrastructure, not by reading docs:

  • ADK's event stream silently returned empty text on Gemini's "thinking mode" responses - parts split across multiple events, and grabbing only the first part missed the real answer.
  • A real network call with no timeout hung a trip for two minutes with the process idle at 0.1% CPU. ADK's own Gemini class builds its client with no timeout set — fixed by subclassing it.
  • Cloud Run silently starves BackgroundTasks of CPU once a request returns, unless you deploy with --no-cpu-throttling — a trip sat at 0/9 steps for 80 seconds before I found that.
  • The heaviest bug wasn't in the app at all: my test suite was quietly writing to the same production Firestore project the deployed dashboard reads from. WARDEN_STUB_MODE only stubs model calls, not storage — and I only caught it because a live transaction count didn't match what was actually in the registry.
  • Vertex AI Agent Engine's enable_tracing=True turned out to be a deprecated flag — it silently sets only one of the three environment variables Google's current tracing docs actually require, so a "configured" observability setup produced nothing in Cloud Trace until I dug into the actual current docs, not the flag's own name.
  • Model Armor only attaches through Vertex AI directly, not the free AI Studio path the live deployment uses — and this project's Vertex AI catalog access tops out at an older Gemini generation than what's deployed, so it stayed a demonstrated, opt-in capability rather than the default.

Accomplishments that we're proud of

  • Held to one rule throughout: verify against the live system, don't assume.
  • That caught real bugs a stub-only demo never would have — the transaction-persistence bug, the Firestore-pollution bug, the deprecated tracing flag — and each one got actually fixed, not just documented.
  • The dashboard's live trace is driven entirely by real backend events, nothing simulated for the UI.
  • The audit ledger is genuinely tamper-evident and survives a restart or a Cloud Run scale-down without losing a step.
  • The Agent Identity and Memory Bank claims aren't from documentation — they're read directly off the deployed resource's own spec.

What we learned

  • The core lesson wasn't about any single API: "configured" and "confirmed" are different claims, and it's worth being disciplined about which one you're making.
  • Nearly every real bug hid behind a reasonable-sounding assumption — that stub mode meant no network calls at all, that enable_tracing actually enabled tracing, that a Console path Google's own docs point to would show what the docs describe.
  • Checking against the live, running system — not just the code or the docs — caught all of it.
  • Came away with a much better feel for ADK's actual runtime behavior: event shapes, planner/thinking-mode quirks, timeout handling.
  • Also learned the operational edges of Cloud Run, Firestore, and Vertex AI that only show up in a real deploy — IAM grants that don't happen automatically, a health-check path Google's own edge intercepts, a model catalog that differs between AI Studio and Vertex AI on the same project.

What's next for Warden - an authorization gateway for agentic fleets

  • Multi-tenant, versioned policy — it's currently a single Markdown document.
  • Harden the reviewer's JSON parsing against a model that ignores the format.
  • Confirm full OpenTelemetry-compliant tracing — still unconfirmed even with the correct env vars set, worth chasing further via the newer Agent Platform runtime console.
  • Bring Agent Identity and Memory Bank into the main live call path — currently a separate, additive deployment because of a real google-adk version conflict, fixable once that pin can be safely bumped.
  • Generalize beyond Travel & Expense — the same gateway pattern (intercept, guardrail, tiered review, live recovery) should apply to any policy-bound agentic workflow a finance or ops team already governs.

Built With

Share this project:

Updates

Submission history