BLACKBOX

• 

  BlackBox Infographic

• 

  BlackBox Architecture Diagram

• 

  BlackBox Image

Inspiration

Companies are already handing consequential decisions to AI agents: approving refunds, reviewing claims, screening applicants. When someone asks why an agent did what it did, most teams cannot fully answer. They have logs of what got called and what came back, not the reasoning behind it, no way to return to the moment before the decision, and no way to answer the question that actually gets asked: would it have done the same thing under last month's rules?

BLACKBOX is the layer that makes those questions answerable. Not a better agent, a recording layer any agent system can sit on top of, plus everything that becomes possible once the recording is complete and impossible to tamper with.

Two things sit at the center of it. The Diary is that recording: an append-only event log, one write method, no update, no delete. The Wiki is condensed working memory built from the Diary, rewritten as facts change, the only thing the agents themselves ever read. Everything else in this project is something the Diary makes possible that would otherwise be a promise, not a fact.

What it does

The Diary sits underneath a six-agent fleet handling a regulated bank complaint workflow, chosen as the proving ground because it is the hardest believable case: statutory deadlines, health information, customers across three jurisdictions. The same recording layer applies to medical triage, insurance claims, lending, or hiring, anywhere a machine's decision is something a person may later have to answer for.

Because every action and its reasoning is written down and never altered:

• Invisible Ink blocks a data leak in a letter that contains none of the sensitive words itself, because a sensitivity tag travels with information through paraphrase and rewriting, not by matching keywords
• The Time Machine rewinds a closed case, changes one governance rule as data rather than code, and replays it to show exactly who a policy change would have affected differently, without touching anything live
• The Eraser retracts a fact and cascades that retraction through every summary derived from it, even several steps downstream, without ever showing a regenerating model the retracted content
• The Stunt Double shadow-runs a candidate agent version against genuine past cases with every write faked, and an independent Gemini judgment blocks promotion if the new version is riskier than what's already live
• The Immune System lets Gemini write its own adversarial attacks against the live agent code, keeps every attack that ever worked in a corpus that only grows, and counts a success only when an actual policy boundary was crossed
• The Crash Test injects faults the agents themselves have to read and react to, so a genuine contradiction between two systems of record can never be retried away, only escalated
• The fleet runs unattended, waking itself on a schedule rather than a button, including suspending for days waiting on a slow external system and resuming with full context rebuilt from the Wiki

How I built it

Google ADK agents, Gemini 3.5 Flash on Vertex AI for every model call, deployed on Cloud Run with Firestore (hot), BigQuery (warm), and Cloud Storage/Parquet (cold) as a three-shelf tier, Pub/Sub and Cloud Scheduler driving every trigger so nothing starts from a manual action. The one rule everything else follows: the event store has exactly one write method, no update, no delete. Firestore writes go through create(), not set(), so an overwrite fails at the database rather than succeeding quietly.

On top of that: a sensitivity-label lattice that propagates through every model call and blocks disclosure by label ancestry rather than keyword matching; a policy-as-data replay engine (CEL expressions) that rebuilds state from the log rather than reading current state, isolated from production by capability, not a flag; a shadow-evaluation system that stubs every write and uses Gemini as an independent judge before promoting a new agent version; a red-team system where Gemini generates adversarial attacks against the deployed agent code, scored strictly by whether a policy boundary was crossed; and fault injection that surfaces faults as tool results the agents themselves read, so a contradiction cannot be retried away.

Challenges I ran into

The build spec calls for "Gemini 3.5 Flash or newer." Every 3.x model id 404s against Vertex AI's regional endpoints, nine variants tried. The fix was not a different model, it was a different endpoint: gemini-3.5-flash resolves on Vertex's global endpoint, not a region, which is easy to miss since the error looks identical to the model simply not existing yet.

A machine-level gcloud TLS failure turned out to be Norton Antivirus intercepting TLS with its own root CA, unrelated to this project but blocking all Google Cloud work from this machine until diagnosed.

The most consequential bug was not a crash. An earlier pass had marked the foundational event-store phase complete on the strength of tests that never actually exercised the write path, only asserted method signatures existed. Finding and fixing that, and rewriting a tiering job that had shipped as a no-op, mattered more than any later phase, because six later phases depend entirely on the recording being trustworthy.

What I learned

Decide what a failure means before building the thing that detects it. The tempting failure criterion, whether the model "sounded" wrong, measures nothing. The honest one, whether a policy boundary was actually crossed, is what caught genuine issues in the red-team system. And prefer capability to discipline: every guarantee in this system holds because the dangerous action is unreachable, no client for a replay to call, no tool registered for an agent that shouldn't have one, not because the code remembers not to take it.

What's next

Move the red-team attack corpus off the Cloud Run instance filesystem to Cloud Storage so it survives revision changes, and add a scheduler job for ongoing automated shadow and red-team runs if continuous testing, rather than on-demand, is wanted.

Built With

• cel-python
• fastapi
• gemini
• gemini-3-5-flash
• google-adk
• google-cloud
• google-cloud-bigquery
• google-cloud-firestore
• google-cloud-pubsub
• google-cloud-run
• google-cloud-scheduler
• google-cloud-trace
• pyarrow
• python
• server-sent-events
• uvicorn
• vertex-ai