AgentMesh

• 

Inspiration

There's a moment that kept replaying in my head while I was scoping this project: an enterprise deploys its first AI agent, it works, everyone's thrilled, and six months later there are eleven of them running quietly in production and nobody in the room can answer a simple question — which one touched the customer database last Tuesday, and who approved that?

That's not a hypothetical failure mode. It's the natural end state of "just ship the agent." Every team can wire an LLM to a few tools and call it done. The much harder, much less glamorous problem is what happens after — when you have a fleet, not a demo, and the question shifts from "does it work" to "can I trust it, prove it, and stop it the moment it doesn't deserve that trust." I didn't want to build another clever agent. I wanted to build the thing that makes a fleet of agents safe to actually deploy.

What it does

AgentMesh is a real, deployed enterprise AI control plane sitting in front of six specialized AI agents — Fraud & Finance, IT & Security, Compliance, Expense Approval, HR Leave, and Legal Contract Review. They all work against a synthetic company I built for this, Northbridge Retail Co., but the infrastructure underneath it isn't synthetic at all: a real GitHub repository that the IT/Security agent actually monitors and files issues against, real Firestore records, and real Cloud Run services doing real work.

Here's the part I think matters most: every enterprise data and tool operation performed by an agent is required to pass through one Gateway. That Gateway follows the same six-stage path every time: Authentication → Identity → Policy & State → Threat Shield → Tool Access → Audit.

It checks who's calling. It checks they're a registered, active agent. It checks policy — including a rule that says, plainly, Finance can never read HR data, no matter what. It scans content for prompt injection, tool poisoning, PII, and secret leakage, using fast pattern matching for known attacks and a real Gemini classifier for novel injection attempts. Only then does it forward the request. And it records the decision and execution metadata through the Gateway-controlled audit path, whether the answer was yes or no.

One more governance layer worth calling out: the Expense Approval agent operates under a real spending policy enforced entirely by the Gateway — a per-transaction cap, a daily limit, and an approval threshold above which requests automatically route into the same human-approval workflow, not a separate system. The agent never judges its own budget; it requests an operation, and the Gateway is the sole authority on whether it's allowed — the same principle as every other check in the pipeline.

I didn't want you to have to trust that description. So the dashboard has a Policy Playground where you can watch an agent actually get denied access to restricted data in real time, and a Threat Shield Playground where you can enter an attack yourself and watch the security pipeline catch it.

The fleet also operates asynchronously. Investigations are queued through Pub/Sub, executed by Cloud Run workers using Google ADK, and persisted as durable workflow state in Firestore. A workflow can pause for human approval and resume later without depending on the original agent process.

How I built it

Underneath the reasoning, this genuinely runs on Google ADK — LlmAgent, FunctionTool, and Runner.run_async(). Gemini 3.5 Flash is the model deciding, during execution, which tool to call next. The FunctionTools expose Gateway-backed operations rather than giving agents direct Firestore or GitHub access. It's not a script that calls functions in a fixed sequence and dresses it up as an agent afterward. The model is actually driving the tool-calling loop.

Everything runs asynchronously through Pub/Sub. You trigger an investigation, it returns immediately, and a worker performs the real work in the background. The worker uses an atomic Firestore claim transaction so duplicate Pub/Sub deliveries cannot cause two workers to process the same queued workflow simultaneously.

And no workflow-critical state depends on process memory. Workflow state, findings, and case context are persisted in Firestore. That's what allows the system to survive a Cloud Run revision change or process restart mid-task. I didn't just design for this — I actually tested it by forcing a live Cloud Run revision change and watching a fresh instance resume the workflow correctly from Firestore.

Challenges I ran into

The infrastructure itself was mostly cooperative. The real challenge, and it showed up again and again in almost the same shape, was trust — specifically, the gap between "this is reported as working" and "this is actually working."

Early in the build, I discovered agent service accounts had direct access to Firestore, despite the architecture supposedly enforcing that only the Gateway could touch the database. An external audit caught it. I didn't just remove the access — I proved it was gone by running an isolated test under each agent's real identity and watching it get a genuine permission-denied error back from Google Cloud itself.

Later, something more subtle: I'd been told the agents had "genuine ADK adoption." What that actually meant, when I finally read the code myself instead of trusting the summary, was that the ADK objects existed in the file but Gemini was still being called directly around them — a Runner that was constructed but never actually run. I only believed it was fixed the second time, after reading two different agents' complete source files myself, line by line, and confirming the tool-calling loop was genuinely what drove execution.

And then there was a race condition that only revealed itself when I stopped assuming and started attacking my own system: I fired two duplicate job messages three milliseconds apart and watched to see what would happen. The first version let both through, silently duplicating work. The fixed version used an atomic Firestore claim, and one request won cleanly while the other was rejected before it ever touched the workflow execution. That test — deliberately trying to break my own concurrency guarantee — taught me more than any code review could have.

Accomplishments that I'm proud of

Every claim in this submission is backed by something I personally reran and watched happen, not something that merely should be true.

The restart survival is real — I forced a live Cloud Run revision change during the workflow lifecycle and watched a fresh instance resume the work correctly, using the state persisted in Firestore.

The Threat Shield's smarter detection is real too. I wrote a phrase deliberately designed to slip past the deterministic pattern matching, and Gemini caught it anyway. The Gemini classifier path took roughly 17× longer than the fast regex path in my live test, which is itself a small but honest signal that a real model call happened rather than a canned response.

The spending policy is also enforced at the Gateway rather than inside the agent. The daily spend limit is computed live from real audit log entries, not a stored counter — and I proved it holds correctly across a real four-transaction sequence in a single day, including a workflow-deduplication safeguard so the same transaction cannot be counted twice.

I'm proud, too, of what I chose not to inflate. The registry contains ten department-level agent manifests. Six are fully real, tested, and working. The other four are explicitly marked pending — not built, and I said so, rather than padding the number to look more impressive than it is.

What I learned

The biggest thing I take away isn't about agents at all — it's about what "done" actually means.

Least-privilege identity sounds like a checkbox until you've watched a system quietly bypass it and have to go prove, with a real denied API call, that it's genuinely closed.

Async correctness sounds simple until you've watched a naive version silently double-process the same job and had to fix it with an atomic transaction instead of a bigger if statement.

And the single most useful discipline I practiced on this whole project was treating every "it works now" — mine, my tools', anyone's — as a claim I hadn't verified yet, not a fact I could file away.

That mindset changed how I built AgentMesh: implementation first, then independent verification, then live evidence.

What's next

The four pending agents in the registry — Sales, Support, Supply Chain, and Talent Acquisition — are the obvious next step, and they're genuinely low-risk to add now because the platform pattern is proven rather than theoretical.

Beyond that, I want to move the audit trail from application-enforced integrity to something with cryptographic tamper evidence, and extend the Threat Shield's tool-poisoning detection to cover more sophisticated poisoning embedded in stored enterprise data and tool responses.

AgentMesh started as an attempt to answer a simple question:

If an enterprise has a fleet of autonomous agents, who is watching the watchers?

This project is my answer: a control plane that makes every agent identifiable, every important action governable, every workflow recoverable, and every decision observable.

Built With

• cloud-trace
• fastapi
• firestore
• gemini-3.5-flash
• github-api
• google-adk
• google-cloud-run
• next.js
• opentelemetry
• pub/sub
• python
• react
• secret-manager
• typescript
• vertex-ai