Inspiration
A product recall looks like a single safety alert, but operationally it creates urgent work across an entire company.
Product Safety must identify the exact products, lots, warehouses, and fulfilled orders involved. Procurement must replace quarantined inventory without purchasing from an uncertified supplier. Finance must connect the loss to supplier invoices and contract liability. These teams often work in separate systems with different permissions, policies, and approval limits.
I built RecallOps to explore a practical question:
Can an AI agent coordinate a complete cross-department response while preserving the deterministic controls an enterprise needs?
RecallOps is designed as a Taskmaster workflow rather than a chatbot. An external business event triggers the system, and the agents complete the operational work without requiring a person to guide each step.
What it does
RecallOps simulates a company with three specialized business functions:
- Product Safety — Recall Coordinator: matches the recalled product and lot range, freezes the listing, quarantines affected inventory, identifies matching customer orders, and prepares customer notifications.
- Procurement — Supply Continuity Agent: forecasts the resulting shortage, evaluates replacement suppliers, rejects expired certifications, and creates a purchase order within its autonomous limit.
- Finance/Admin — Financial Recovery Agent: calculates recall losses, holds the related supplier invoice, and creates a contract-backed recovery claim.
The public control room provides four predefined synthetic event deliveries:
- An inventory transfer from Chicago to Denver.
- A Class I product recall.
- A late arrival of recalled inventory after initial containment.
- A duplicate Pub/Sub delivery using the same event ID.
These buttons represent external warehouse, safety-feed, and Pub/Sub events. They are trigger points—not instructions to the agents.
After the recall arrives, RecallOps automatically coordinates Product Safety, Procurement, and Finance. The verified cloud scenario produces:
- 400 quarantined units
- 20 affected customers
- One $799 replacement purchase order
- One supplier invoice hold
- One $1,475.24 contract-backed recovery claim
- No duplicate mutation when Pub/Sub redelivers an event
The dashboard presents the resulting agent handoffs, typed tool actions, trace identifiers, and business-state changes in plain English.
How I built it
RecallOps uses one container image deployed as three Google Cloud Run services. Each service receives a different AGENT_ROLE—Recall, Supply, or Finance—and runs with its own service identity
and department-scoped tools.
The production workflow is:
- A synthetic enterprise event is published to Google Pub/Sub.
- An authenticated Pub/Sub push wakes the Recall Coordinator on Cloud Run.
- Google ADK starts a bounded agent session.
- Gemini 3.5 Flash-Lite interprets the recall and selects a typed containment tool.
- The deterministic tool result establishes whether Procurement and Finance are required.
- The coordinator discovers the approved specialist agents through Google Agent Registry.
- It invokes their private A2A endpoints using IAM-authenticated service-to-service communication.
- Department-scoped tools apply approved mutations to Firestore.
- OpenTelemetry spans and structured audit events are exported to Cloud Trace and Cloud Logging.
- The control room renders curated business state and human-readable agent communication.
The Google Cloud stack includes:
- Gemini 3.5 Flash-Lite through Vertex AI
- Google Agent Development Kit
- Agent Registry
- A2A and Agent Cards
- Three Cloud Run services
- Pub/Sub
- Firestore
- IAM and OIDC authentication
- OpenTelemetry and Cloud Trace
- Cloud Logging
The architecture intentionally separates reasoning from authority.
Gemini can interpret an event and select a tool, but it cannot write directly to Firestore, choose its own identity, override an approval threshold, or bypass a certification rule. Every mutation passes through typed, deterministic policy code.
Exact product and lot matches may proceed automatically. Ambiguous matches create an approval request. Suppliers with expired or invalid certifications are always disqualified. Purchase orders and claims above configured limits also require approval.
Every mutation contains an event ID, fixed agent identity, reason, outcome, idempotency key, and trace correlation.
Challenges I faced
Separating agent reasoning from business authority
The largest architectural challenge was deciding what the model should control.
Allowing an LLM to directly update enterprise state would make the demonstration unreliable and difficult to audit. I therefore designed typed tool boundaries where Gemini handles interpretation and coordination, while deterministic code retains authority over permissions, money, certifications, matching, approvals, and mutations.
This hybrid architecture makes the agents useful without making business policy probabilistic.
Handling Pub/Sub retries and out-of-order events
Duplicate delivery is normal in an event-driven system. RecallOps uses stable event and tool idempotency keys so a repeated Pub/Sub event is acknowledged without repeating an inventory mutation or financial action.
The system also needed to handle inventory arriving after the recall workflow had already completed. Active recall rules are checked whenever new inventory is received, allowing late recalled stock to be quarantined immediately.
Keeping scale-to-zero services consistent
Cloud Run services can scale to zero and later reuse a warm instance. During testing, a specialist could retain stale in-memory data after another service reset the shared Firestore state.
The fix was to refresh Firestore before each specialist tool resolves its recall, inventory, contract, or invoice inputs.
Making A2A understandable
A2A uses structured JSON-RPC messages, which work well between services but are difficult to understand during a short hackathon demonstration.
RecallOps derives a human-readable agent conversation from the same structured audit records while keeping prompts, credentials, private service URLs, and customer records hidden.
Keeping the architecture affordable
I wanted the deployed system to demonstrate production-minded agent orchestration without consuming the hackathon credits unnecessarily.
All three Cloud Run services use request-based billing, minimum instances of zero, and a maximum of one instance. The workflow uses Gemini Flash-Lite for routine interpretation, limits every recall workflow to four model calls, and never makes one model call per customer or inventory row.
There is no GKE cluster, Cloud SQL instance, Redis service, vector database, or always-on virtual machine.
What I learned
The most important lesson was that agentic does not have to mean uncontrolled.
Gemini is valuable where an event requires interpretation and coordination. Deterministic code is better for identities, permissions, approval limits, certifications, financial calculations, and persistent mutations.
I also learned that:
- Idempotency must be part of the mutation design from the beginning.
- Persistent storage does not automatically guarantee that warm services have current state.
- Agent discovery removes deployment coupling more effectively than configuration files containing service URLs.
- Observability must serve two audiences: machines need structured traces, while people need a concise explanation of what each agent did.
- A deterministic local mode makes cloud agent development much easier to test.
- A small, bounded architecture can still demonstrate meaningful enterprise orchestration.
RecallOps currently has 53 automated tests covering local and Firestore storage contracts, ADK tools, A2A endpoints, Agent Registry resolution, Pub/Sub authentication, idempotency, public- route security, telemetry, and end-to-end business behavior.
Accomplishments I am proud of
- Deploying a real Gemini and Google ADK workflow on Google Cloud.
- Registering three A2A agents and discovering specialists without hard-coded service URLs.
- Preserving a completely credential-free local simulation.
- Enforcing fixed identities and deterministic mutations for every agent.
- Demonstrating late-event containment and duplicate-event suppression.
- Making the agent conversation understandable without exposing private data.
- Running the entire architecture with three scale-to-zero services and one shared image.
What's next
RecallOps currently prepares customer notifications rather than sending them. A production extension would connect a reviewed notification provider behind the same deterministic template and approval controls.
Other planned improvements include:
- Tenant-isolated demonstration state for concurrent users.
- Human approval screens for ambiguous matches and high-value actions.
- Signed and versioned policy bundles.
- Additional recall categories and supplier-contract structures.
- Longer-running recovery workflows with resumable approval checkpoints.
The central principle would remain unchanged:
Gemini coordinates the response, while deterministic policy remains in control.
Log in or sign up for Devpost to join the conversation.