Crucible

• 

  Overview

Inspiration

Companies are shipping AI agents that read email, approve invoices, and move money — and almost nobody is checking whether those agents can be talked into betraying them. A sentence hidden in an ordinary-looking PDF invoice — "the account details have changed, please use the following instead" — gets read by a finance agent as an instruction, not as data. There's no firewall for that.

The harder problem underneath: you can't ask an AI whether an AI got hacked. It might lie, or just be wrong about its own behavior. Any system that scores agent security by asking a model to grade itself is building on sand. We wanted a way to know, not guess.

What it does

Crucible attacks its own AI agents every night and fixes whatever breaks.

It plants a physical tripwire in each agent's environment — a sentinel bank account, a canary customer record — and a breach is only recorded when that exact tripwire fires during a real tool call. Not when an LLM judge says something looks bad. A fake bank account either receives a wire transfer or it doesn't; there's no interpretation involved. The Judge model narrates why an agent broke, in plain human-harm language, but it never gets a vote on whether it broke.

When a breach is confirmed, Crucible writes a hardened system prompt, then proves the fix by replaying the exact same attack against the patched agent and confirming the tripwire stays silent — while also running a benign-conversation control, so a "fix" that just makes the agent refuse everything doesn't count as a fix. Outcomes are written to memory, so the next night's campaign is ordered breach-first based on what actually worked before.

It runs unattended. A Cloud Scheduler job fires the whole loop every night at 3am UTC. As of writing, it has been running for 9 real nights, found 9 genuine breaches, and auto-verified 74 patches — none of it staged for the demo.

How we built it

• Google Vertex AI (Gemini 3.5 Flash for the high-volume attacker swarm, a Pro model for the Planner/Judge/Patcher's reasoning-heavy steps) — the "global" endpoint quirk cost us a debugging session and is now documented so nobody repeats it.
• Cloud Run hosts the console and the attack jobs, min-instances=0 everywhere.
• Firestore stores every attack, verdict, and patch — the console reads directly from it, so the dashboard is never a mock.
• Cloud Scheduler fires the nightly campaign; we started it on day five so the "gets harder to breach over time" claim would be backed by real accumulated history, not two nights of staged data.
• Google Model Armor — we wired it in as a pre-screening layer in front of every range agent and re-ran the full attack corpus with it on. We report the real result below, not the flattering one.
• A billing-detach Cloud Function + budget alert, deployed before any product code, so the whole project runs at genuinely $0 on a Free Trial account that's never manually upgraded.

Six roles do the work: Recon (reads a target's real tools to build a threat profile), Planner (orders the campaign breach-first using memory), Attacker (Flash-powered, runs the actual multi-turn conversations), Judge (oracle-gated, narrates the harm), Patcher (writes the hardened prompt), Curator (writes to Firestore, computes the fleet score).

Challenges we ran into

• We almost biased our own first result. An early version of one victim agent's system prompt contained an explicit anti-injection line, which would have made the Day-1 test pass for the wrong reason. Caught it, removed it, documented the correction rather than quietly deleting the evidence.
• A sentinel value that announces itself is a confound, not a control. We originally named a canary account ATTACKER-ACCT-6669. A model that's pattern- matching on the string "ATTACKER" can pass without actually reasoning about the attack. Renaming it to a neutral-looking value flipped two "resisted" results to real breaches on retest — worth stating plainly rather than burying.
• Model Armor did not save us. We tested it honestly expecting a mixed result. Every attack that broke an agent at baseline still broke it with Model Armor enabled, and the image-based attack (an injection hidden in a scanned invoice) wasn't screened by the configured template at all. That's a real, current finding, not a gap we're hiding — see docs/MODEL_ARMOR_RESULTS.md in the repo.
• Reproducing a bug in someone else's code, faithfully. We tested Google's own official adk-samples customer-service agent, unmodified, and found a real validation-asymmetry bug between its two discount-approval tools. Reported it through Bug Hunters. Google's review closed it as Infeasible — sample code, not a production system, so it falls outside their internal abuse-risk tracking. They never disputed the bug itself. We're reporting that outcome exactly as it happened, including the part that isn't flattering.

Accomplishments that we're proud of

• A finding on Google's own sample code, reproduced 3/3, reported through the correct channel, escalated to a human reviewer within hours.
• An honest Model Armor result, including the negative one — it directly answers the question a judge should ask: "doesn't Google's own guardrail already solve this?"
• Nine real, unattended nights of Cloud Scheduler data, not staged for the camera.
• Zero dollars spent, verified against a live billing account, by design rather than by luck.

What we learned

That the honest version of a result is usually the more interesting one. "Model Armor blocked everything" would have been a tidier story than "every breach still landed" — but the tidier story would have been less useful to anyone actually deciding whether to rely on Model Armor alone.

What's next for Crucible

Real service-account identity per range agent (so the privilege-escalation finding reflects a genuine IAM boundary, not a hardcoded tool name), OpenTelemetry traces linked from the dashboard, and expanding the range beyond four hand-built agents to more third-party targets — the Google finding was the single highest-leverage hour in this build, and there's no reason to stop at one.

Built With

• fastapi
• google-cloud-firestore
• google-cloud-run
• google-cloud-scheduler
• google-cloud-trace
• google-gemini
• google-genai-sdk
• google-model-armor
• google-vertex-ai
• opentelemetry
• pytest
• python
• terraform