Multi-Tenant Multi-Agent Customer Support Fleet

Inspiration

Almost every e-commerce site has a chatbot, or feels like it should. Yet most of them are still frustrating, because “chatbot” usually means an FAQ layer sitting beside the real shopping experience, not something that can actually help a customer complete what they came to do.

A chatbot can explain a return policy, but it cannot process the return. It can suggest that you check your order, but it sends you somewhere else to find it. Customers still have to jump between keyword search, product pages, carrier websites, support forms, and phone lines just to complete a single journey.

We wanted to remove those handoffs.

Our vision was an AI shopping and support experience where the conversation itself becomes the interface to the store. A customer should be able to type or speak naturally and, from the same conversation, discover products, ask questions that keyword search cannot answer, compare alternatives, place an order, check out, track a shipment, get support, and request a refund, without being pushed from one system to another.

But the moment an AI agent can act, not just answer, the problem changes completely.

Looking up a product is low risk. Moving money, modifying an order, or issuing a refund is not. An agent that can take real actions on behalf of customers needs more than a good prompt telling it to “be careful.” Authorization, business rules, validation, auditability, observability, and clear boundaries have to be part of the architecture itself.

Refunds sit at the intersection of customer experience, money, policy, security, and operational risk. If we could make an autonomous refund workflow both effortless for the customer and trustworthy for the retailer, we would be solving one of the hardest versions of the problem, not just building another conversational demo.

We also wanted to go beyond a system built for one fictional store. From the beginning, we designed it as a multi-tenant product that multiple retailers could run on while keeping their data, configurations, policies, customers, and agent actions safely isolated from one another.

The inspiration ultimately became simple:

What if e-commerce support stopped being a chatbot that tells you what to do, and became an intelligent system that could actually do it with you, end to end, safely?

That is the gap we set out to close.

What it does

A multi-tenant AI commerce and customer-support platform that lets shoppers search, compare, buy, track, return, and get help through one continuous conversation, while giving merchants structural controls over identity, data isolation, agent actions, and money movement.

Two ways in, one experience

A standalone web app, or an embeddable widget any merchant drops onto their own storefront with a single <script> tag.
Customers type or speak; voice input works on both surfaces.
Identity arrives one of two ways: the merchant hands off an already-authenticated customer with a signed assertion, or the customer signs in inside the chat through a popup served from our own origin. Either way the conversation continues exactly where it left off, with no page navigation and no second account.

Shopping

The solution turns the conversation into a complete shopping interface with:

Semantic (RAG) product search with an automatic keyword fallback, so a search never returns empty just because the phrasing did not match the catalog.
Product details, live inventory, and real customer reviews, returned together in one answer by default, or individually when the customer asks for only one.
A real agent-assisted basket: add, view, update quantity, remove, all reachable by simply asking, and mirrored by five REST endpoints for the page's own basket UI.
Checkout renders an order card inside the chat, address pre-filled from the customer's last order, with Confirm disabled until the address is complete. A returning customer usually just clicks Confirm.
The agent only ever stages an order. Nothing is purchased until the customer's own click, and any basket change supersedes an outstanding proposal so a stale card can never place the wrong order.

Orders

Customers can ask naturally for : Order history, order status, and order details.
Guest order lookup for a customer with no account, step-up verified by order number plus email, capped at three attempts, and scoped to exactly that one order. It never unlocks billing or refunds.
Live carrier tracking through two independent integrations: a governed MCP server for FedEx, and a direct carrier API for everything else. The agent answers "where is it right now" without sending anyone to a tracking site.

Billing

Invoice lookup by invoice or by order, full invoice history, payment status, and payment history, on demand, in the same conversation.

Refunds

The agent walks the customer through eligibility, reason selection from a fixed taxonomy, a retention offer (a real incentive to keep the item, offered before the refund proceeds, not after), photo evidence for damage or defect claims, and refund-method choice.
It can tell the customer which items on an order actually qualify before they start.
The agent can only ever stage the request. A second, different human approves it on a staff-facing dual-control queue, where the requester can never be the approver, before any deterministic backend code touches Stripe or a store-credit balance.
After approval, the same chat issues the RMA and return instructions, accepts the customer's return tracking number, and answers status and evidence questions on a refund already in flight. No portal, no phone call, no form.

Continuity and accountability

Cross-session memory: the agent registers what it learns about a customer as they talk, such as how they prefer to be refunded, so weeks later, in a brand new conversation, it already knows. Verified live, not scripted.
Structural escalation: a frustrated customer, or one who simply asks for a human, is short-circuited onto a real staff queue by a plugin that fires ahead of the model call, rather than being left to the model's own judgment about when to give up.
Token-by-token streaming, so answers appear as they are generated.

Underneath: all seven platform pillars, each load-bearing

Pillar	What it actually does here
Agent Registry	Runtime discovery of the refund service; backs the gateway's default-deny list
Agent Runtime	Hosts two separately-deployed resources: the root fleet and the refund workflow
Memory Bank	The cross-session recall above, via two distinct mechanisms
Agent Identity	Zero-trust identity on both resources, no shared service-account key
Agent Gateway	Single default-deny egress chokepoint for everything the fleet reaches outside itself
Model Armor	Screens every inbound message and every tool output
Agent Observability	Traces the full reasoning chain, not just HTTP spans

And it proves itself, continuously

No change reaches a customer without clearing a five-stage evaluation gate: tool tests, single-agent eval, multi-agent eval, post-deploy eval against the live deployment, and a real-traffic canary check against the previous release. A release only takes production traffic after beating an absolute quality bar and not regressing against what is already running. That promotion runs through real dev, staging, and prod environments provisioned from the same Terraform. Once live, every turn is traced, structured logs and Prometheus metrics ship continuously, a circuit breaker isolates a failing dependency, and a watchdog outside the agent's own event loop replaces a worker that has gone silently unresponsive, so a fault degrades gracefully instead of hanging a customer mid-sentence.

How it all fits together

A single question, "where is my order, and can I return it?", touches almost every part of the system. Following it end to end is the clearest way to see how the pieces integrate:

The customer types it into the widget, embedded on the merchant's storefront. The widget resolves which tenant it belongs to from its own script tag, and sends the request with the identity the merchant handed off.
The FastAPI backend on Cloud Run validates that token, checks the request's origin against that tenant's allow-list, resolves the tenant to its own Firestore database, and screens the message through Model Armor before an agent ever sees it.
The escalation plugin runs first, ahead of the model, and would divert the turn to a human queue right here if the customer were frustrated. They are not, so the turn proceeds.
Memory Bank preloads what we already know about this customer into the system instruction, both the natural-language history and the structured profile.
The root agent routes to the Order specialist as an in-process AgentTool. It verifies ownership of the order, reads it through the CommerceProvider abstraction from that tenant's database, and calls out for live carrier status through the FedEx MCP server, which is reachable only because Agent Gateway allows that one registered destination and denies everything else by default.
"Can I return it?" is a different kind of problem, so the root agent calls the refund service over A2A, again through the gateway. That service is a separate Agent Runtime resource with its own Agent Identity, running a graph workflow that checks eligibility against versioned policy and then genuinely pauses to make a retention offer.
The customer replies hours later, possibly routed to a completely different worker. Agent Platform Sessions plus a Firestore task store carry the workflow's state, so it resumes mid-flow rather than starting over.
At the end, the workflow stages a refund request. It cannot execute one; that function is unreachable from anywhere the LLM can call. A second human approves it on the staff queue, inside a Firestore transaction that also enforces that the approver is not the requester, and only then does money move.
Every step of that, across both resources, is traced through OpenTelemetry, and the whole path is re-verified on every release by the evaluation gate before any of it reaches production traffic.

The integration story is that none of those layers is optional decoration. Remove the gateway and the MCP call stops being governed. Remove Sessions and step 7 breaks. Remove the transaction in step 8 and dual control becomes advisory. They were built to hold each

How we built it

Agent orchestration: collaborative agents and a graph workflow

Built on Google ADK, using two different orchestration patterns on purpose, because they do not compose into a single deployed agent and we chose to split the system rather than force a fit:

Collaborative agents for Product (8 tools), Order (12, or 14 with both carrier integrations enabled), and Billing (13). A root coordinator routes each request to the right specialist as an in-process AgentTool, all on Gemini 3.5 Flash. This fits anything that resolves within one turn, because the tool call returns inside the same exchange the root agent is already handling.
A graph workflow (ADK 2.0's Workflow API) for refunds only. Refunds do not resolve in one turn: the flow needs a genuine human-in-the-loop pause for the retention offer, another for evidence, and it must resume later, possibly hours later, possibly on a different worker. That requires the workflow to own its own session lifecycle, which a nested AgentTool call cannot give it, because the root agent's turn has no way to suspend mid-tool-call and hand control back later without eplaying the whole exchange. So the refund workflow is its own separately-deployed Agent Runtime resource with its own Agent Identity, reached over A2A rather than nested as a tool.

State is carried by VertexAiSessionService (Agent Platform Sessions) plus a Firestore-backed task store.

Security

Model Armor at two layers: the backend screens every chat message for prompt injection, jailbreaks, and PII before an agent sees it; a second ADK-plugin layer screens every tool's output for tool-poisoning.
Agent Identity, zero-trust, on both deployed resources, with no shared service-account key between them.
Agent Gateway in Agent-to-Anywhere (egress) mode as the single chokepoint every outbound call passes through, default-denying any destination not explicitly registered.

Governance

Agent Registry backs that default-deny policy and lets the root agent discover the refund service dynamically at runtime instead of from a hardcoded resource name.
Multi-tenancy is structural, not a filter: every request carries a tenant_id resolved to its own Firestore database, with no default tenant anywhere, including in the widget, which resolves its tenant at runtime from its script tag.
Dual control on refunds, enforced server-side against a role on the approver's own user record, inside the same Firestore transaction that writes the refund.
Merchant identity handoff via signed HS256 assertion, its per-tenant secret sealed under an AES-GCM master key, with the assertion's lifetime capped regardless of the expiry it claims for itself.

Memory

Two distinct Memory Bank mechanisms, not one:

A natural-language topic, captured and semantically searched through ADK's stock preload tool.
A structured profile (preferred refund method, open-escalation flag) retrieved by scope-keyed lookup through a preload tool we wrote ourselves, because the stock tool only reaches semantic search, never structured retrieval.

Both are written by the same after-turn callback and read back before the model call on every later turn, which is what makes recall weeks later real rather than staged.

Escalation

Runs as an ADK plugin, not a callback on any single agent, so it fires ahead of the model call, before memory search or inference cost anything, and covers every specialise rather than only the root agent.

Observability

Every turn traced end to end through OpenTelemetry, spans included.
Structured logs and Prometheus metrics shipped continuously via a logging plugin.
Agent Gateway independently generates network-layer telemetry for every call crossing it.

Evaluation

Six test layers run before every deployment; four are free and run on every pull request. The two that call a model support switchable profiles, so a PR gets a cheap check and a merge to main gets a full LLM-judge rubric pass.
ADK's eval SDK silently breaks on multi-agent AgentTool handoffs during inference, so the harness drives the deployed engine directly rather than trusting the built-in runner.
Post-deploy eval plus a real-traffic canary check against the previous champion both gate production traffic, and a canary quality check keeps re-scoring real sessions daily after release.

AgentOps

A circuit breaker (closed, open, half-open) trips after five consecutive failures,with **exponential backoff* from one second to sixty.
A watchdog thread outside the event loop force-kills a worker whose turn exceeds 90 seconds, because some calls under a specific mTLS condition ignore cooperative cancellation entirely in Agnet Runtime.
A bad refund release rolls back to any prior commit's exact code and deploy logic via a temporary git worktree, not merely a container image tag.

CI/CD

Infrastructure is Terraform, applied identically across dev, staging, and prod.
One main branch, promoted by git tag through Cloud Build: a PR runs a fast free check, or a full integration eval with an LLM judge when agent code changed; a push to main deploys to dev; an -rc.* tag deploys to staging with load tests; a version tag ships prod.
A prod release deploys as a shadow revision taking no traffic, then must clear the eval gate and canary check before reaching even a 10% slice.
The refund service, which updates in place and has no shadow revision, deliberately deploys after that gate, so untested refund logic never reaches all live traffic.

Note: this was tested sepaaratly and ommited from the repo demo to save money.

Frontend, widget, and data

React, TypeScript, and Vite, with responses streaming over Server-Sent Events, including a retry frame so a client can discard a stalled attempt's partial text rather than splicing two answers together.
The widget is a second build target from the same codebase: one <script> tag paints a launcher instantly, then lazy-loads the real bundle into a shadow root on the merchant's page, giving genuine first-party storage instead of an iframe's restricted third-party storage.
Two layers of origin authorization gate it, permissive CORS reflection plus a strict per-tenant allow-list, and a mismatched origin gets the same generic error an unknown tenant would, so it cannot be used to enumerate tenants.
Firestore with vector search is the only datastore, reached exclusively through a CommerceProvider abstraction, so swapping in a different commerce backend (a Shopify stub already exists) is configuration rather than a rewrite.

Challenges we ran into

Making an agent that can act without making it exploitable

The whole premise is an agent that does things rather than answers questions, and the moment that's true, the interesting problem stops being conversation quality and becomes this: what stops a sufficiently clever message from causing an unauthorized refund?

Prompt instructions were never going to be the answer. Neither was a confirmation step th model itself decides to show. What we landed on is that the model must be structurally incapable of the dangerous action: the functions that execute a refund live behind a graph the LLM cannot invoke, reachable only after a second human approves inside a database transaction that itself enforces approver is not requester. Getting there meant redesigning the refund flow three times, because each earlier version had a path where a sufficiently persuasive conversation could still reach money.

A workflow that has to survive being interrupted for hours

A refund conversation genuinely pauses. The agent makes a retention offer and the customer replies tomorrow, on a different machine, after the worker that asked the question is long gone. That is a fundamentally different execution model from a chat turn, and it is why refunds ended up as a graph workflow in a separately-deployed service rather than another in-process specialist.

The platform's own suggested path for composing those two models is RemoteA2aAgent, which wraps a remote agent so it looks like a local sub-agent. It cannot survive a redeploy: it holds a live httpx.AsyncClient in a pydantic field, which is not deep-copyable, while Agent Engine's update path deep-copies the entire agent graph before every deployment. Two platform behaviors that each make sense alone are mutually incompatible, and the failure surfaces at deploy time rather than at design time. We replaced it with a plain async function tool speaking A2A directly, which is now the more robust integration anyway because it holds no connection state at all.

Governance that locks you out of your own system

Agent Gateway's default-deny egress is the right security posture, and turning it on broke the agent's ability to reach its own platform APIs. Enforcement denied internal aiplatform.mtls.googleapis.com and Firestore traffic, not the third-party MCP call it was meant to govern, because the gateway denies any unregistered destination regardless of the authorization policy's enforcement mode. Making it safe to enable meant pre-registering 13 internal Google API hostnames across 65 URL variants first. This is the kind of thing you only discover by turning it on against a live deployment.

Related, and equally non-obvious: Agent Identity's SPIFFE/mTLS tokens are not valid Cloud Run OIDC invoker tokens. Calling our own MCP server returned 401 rather than 403, which reads like a broken credential rather than a wrong kind of credential, and cost real time to diagnose. The order agent now impersonates a dedicated, narrower service account for exactly that one hop.

Failure modes that ignore the language's own guarantees

Under a specific mTLS condition, some outbound calls in the runtime do not honor asyncio cancellation at all, so a correctly written asyncio.wait_for() still hangs forever. Every cooperative timeout we added was defeated by code that never yields control back to check whether it should stop. The only thing that reliably works is non-cooperative: a watchdog thread outside the event loop that calls os._exit() when a turn exceeds its ceiling, because a wedged worker cannot be asked politely to stop being wedged. It's a blunt instrument, deliberately, and the platform replaces the killed worker automatically.

Tooling that fails quietly rather than loudly

Two of these cost the most time because nothing errored. ADK's evaluation SDK produces plausible-looking but wrong inference results on multi-agent handoffs, so the harness now drives the deployed engine directly. And Agent Runtime re-projects its own Agent Registry resource with a fresh random ID on every redeploy of an A2A-capable agent, meaning a registration that worked yesterday silently stops applying today. The deploy script now re-runs registration automatically, every time, because a one-time setup step is a guaranteed future outage.

Accomplishments that we're proud of

An agent that handles money and cannot be talked into misusing it. Not "we told it not to" but "the code path does not exist for it." That property survives any prompt injection, any jailbreak, any future model swap, because it isn't a model behavior at all.

A conversation that survives infrastructure failure. A refund paused mid-flow resumes correctly on a different worker after the original one is gone. Verified live against a real multi-worker fleet, not asserted from documentation.

All seven platform capabilities genuinely load-bearing. Not one demoed and six name-dropped. Remove any single one and something concrete breaks, which we can point at line by line.

A release pipeline that can refuse to ship. A regression cannot reach full production traffic, because promotion requires clearing an absolute quality bar and beating the previous release on real traffic, automatically, with no human in a position to wave it through.

A customer journey with no dead ends. Search, compare, buy, track, return, refund, and escalate to a human, all inside one conversation, on the merchant's own storefront, without a form, a portal, a phone call, or a second app anywhere in it.

What we learned

Safety properties are architecture, not instructions. Every guarantee here that actually holds is one we enforced structurally, through unreachable code paths, transactional constraints, and default-deny networking. Every guarantee we initially tried to express as a prompt instruction eventually had to be rebuilt properly. The structural version was also, consistently, less work than the prompt version plus the incident it would have caused.

Design for the interruption, not the happy path. Treating "this conversation will be abandoned mid-flow and resumed on different hardware" as the normal case rather than the exception is what forced the refund flow into a separate resumable service, and that decision is why the feature works at all rather than only demoing well.

Silent failures cost more than loud ones. The bugs that consumed the most time were the ones where something returned successfully with the wrong result: an eval SDK yielding plausible wrong scores, a stale registration that no longer applied, a hung call with no exception. We now instrument for absence, not just for errors, because "nothing happened" is a failure mode too.

What's next for Multi-Tenant Multi-Agent Customer Support Fleet

Onboarding a real merchant. The multi-tenant foundation is built and isolated; implementing the CommerceProvider interface against a live Shopify store rather than the existing stub is the step that turns this from a system that can serve many retailers into one that does.

Proactive support instead of reactive. Everything today starts with a customer asking. The same data and the same escalation machinery could reach out first: a delayed shipment detected before the customer notices, a failed payment resolved before it becomes a support ticket.

Deeper voice. Voice input works today, but the natural end state for "I don't want to browse a website" is a full voice conversation, which the streaming architecture already mostly supports.

Built With

a2a
agent-gateway
agent-identity
agent-observability
agent-registry
agent-runtime
agent-runtime-session
cloud-build
cloud-run
fastapi
firestore
gemini
gemini-enterprise-agent-platform
google-adk
google-cloud
mcp
memory-bank
model-armor
opentelemetry
python
rag
react
terraform
typescript
vector-search