Agent BlackBox — governing AI agents before they act

Inspiration

AI agents are crossing a line: from suggesting actions to taking them — spending money, approving invoices, moving inventory. The hard part is no longer making an agent capable. It's making it safe to let loose on real systems. A single hallucinated purchase order or a missed approval isn't a bad answer anymore — it's a business incident.

Most "agent safety" lives inside the prompt, where it can be argued away. We wanted the opposite: a governance layer the agent cannot talk its way past, that sits between intent and execution, records everything, and remembers. So we built Agent BlackBox — a domain-agnostic governance runtime for action-taking agents — and drove it with a real business system, ProcurementOps (full procure-to-pay), operated by an autonomous Gemini agent.

What it does

A Gemini agent runs procurement as an employee would. When it tries to commit spend, Agent BlackBox evaluates the action before it executes. Unsafe actions are blocked; the agent treats the block as information, remediates the missing controls, and retries through the governed path. Every step lands on an audit timeline, MongoDB Atlas remembers it as semantic memory, and Gemini can explain any decision — grounded in similar past cases.

Conceptually, an action \( a \) executes only when it clears every layer:

$$ \text{execute}(a) \iff \text{auth}(a)\;\wedge\;\text{role}(a)\;\wedge\;\bigwedge_{c\,\in\,C(a)}\text{satisfied}(c) $$

— authentication, role (segregation of duties), and all required controls \( C(a) \). Auth decides who may act; governance decides whether the action is safe.

How we built it

• The agent — a single Gemini (3.1 Pro) LlmAgent built on the Google Agent Development Kit (ADK). Its persona is its only program — no hardcoded workflows; it selects tools per task.
• The governed boundary — the agent acts only through ~51 Model Context Protocol (MCP) tools, each a thin proxy to a real ProcurementOps HTTP route. There is no privileged path: authentication, role-based access control, and Agent BlackBox governance apply to every call. A 403 comes back as structured data, so the agent can remediate instead of crashing.
• Memory — MongoDB Atlas stores agent episodes, governance decisions, and controls. Past cases are embedded with Gemini text-embedding-005 and retrieved by Atlas Vector Search using cosine similarity,

$$ \text{sim}(q, m) = \frac{q \cdot m}{\lVert q\rVert\,\lVert m\rVert}, $$

so Gemini's explanations are grounded in the most semantically similar prior decisions — read-only; Agent BlackBox stays the execution authority.

• Stack — FastAPI + Python backends, a Next.js / React / Tailwind console, all containerized and shipped as four Google Cloud Run services (Cloud Build, Artifact Registry, Vertex AI).

What we learned

• Governance has to be structural, not promised. The only guarantees that held were the ones the agent couldn't reach around — enforced at the route, not the prompt.
• Make memory earn its place. MongoDB became valuable the moment it was the agent's semantic memory (vector-grounded explanations), not just a row store.
• An agent that gets blocked is more impressive than one that never is — if you can show it reading the block and remediating. Surfacing that loop was the difference between a checkbox and a story.

Challenges we faced

• Closing every side door. We found the purchase-order lifecycle could advance an unsafe PO to approved, bypassing the gate. Fixing it meant a hard rule: every commit point must be governed, with no ungoverned route to execution.
• Segregation of duties — for the AI too. A single identity must not both initiate an action and satisfy its own controls. We gave the agent its own role-scoped key, so SoD constrains it exactly like a human (it gets 403 forbidden_role and must route around it legitimately).
• Making the agentic layer visible. It started as a CLI. Judges needed to watch Gemini act, so we built an Agent Console that streams the live governed trace — blocked steps, remediation, commit — and unwrapped ADK's MCP response envelope to classify governance blocks correctly.
• Production realities: Cloud Run cold starts on the first Gemini call, dynamic Atlas IP allowlisting, and keeping Agent BlackBox strictly generic (it must never import the business system) under a one-way dependency boundary.

The result: an agent that does real work, gets stopped when it shouldn't act, fixes the problem, and commits — with the whole decision remembered and explainable. Agents act. Agent BlackBox decides. MongoDB remembers. Gemini explains.

Built With

• atlas-vector-search
• atrifact-registry
• cloud-build
• docker
• fastapi
• google-adk
• google-cloud-run
• google-gemini
• google-genai
• httpx
• model-context-protocol
• mongodb-atlas
• next.js
• pymongo
• python
• react
• tailwind-css
• typescript
• uvicorn
• vertex-ai