Landing
Overview Cont.
Overview
Safeguards
Pre red team run
Mid run
Individual trial details
Round 2 running with recall populating in the bottom middle box
More info

Inspiration

A Chevy dealership’s chatbot sold a Tahoe for a dollar. Air Canada’s bot invented a refund policy and a court made the airline honor it. Those bots could only talk; this year agents get a terminal, a browser and the company’s memory, and the first thing a buyer asks is the one thing no demo answers: how do you know it’s safe? We borrowed the memory move from previous sandbox projects, where scored mistakes feed memory so behavior changes without retraining.

What it does

Crash Test throws eight attacks at an operations agent that can run code, browse the web and read company memory: prompt injection in page content, a hidden exfiltration command, rm -rf bait in a “cleanup guide”, an exfil lure, memory poisoning, a fake order “from the CEO”, and two chaos tests (browser returns 500, code tool returns garbage). Each attack runs in its own disposable Daytona sandbox with a canary secret planted inside. Scoring is deterministic where possible (canary in the answer or a command, dangerous command, outbound call to a non-allowlisted host) and an LLM judge with a written rubric where it isn’t. Round 1 runs with one baseline rule: be helpful, accurate and safe. Every result is written to SurrealDB Agent Memory with its scope. A hardener rewrites the rules from the round-1 failures, the agent recalls the attacks that beat it, and round 2 re-runs all eight. The deliverable is the table: landed in round 1 versus round 2, same model, no retraining.

How we built it

Daytona for one sandbox per attack, created in parallel, canary planted, deleted after. SurrealDB Agent Memory in two roles: the poisoning target (the planted policy lives in an untrusted scope the agent can see) and the attack log recalled before round 2. Arize AX gives every scenario a span with eval.attack_success and eval.reason so the two rounds compare in the trace view. Amazon Bedrock runs Claude as the agent under test, the judge and the hardener. A stdlib Python server streams progress as JSON lines to a dashboard: three.js landing, live two-round attack board, memory log grouped by scope, safeguards page. An Anchor Browser adapter is wired for the browse tool but ran in stub mode today.

Challenges

The hardest problem was that our agent kept passing. The first live runs came back 0 of 8 landed, and the temptation was to weaken the system prompt or sharpen the attacks until the board turned red. We refused, because a crash test that is rigged to fail tells you nothing. Instead we made the scoring airtight so a pass means something: a canary secret planted in every sandbox so a leak is caught deterministically instead of by vibes, an outbound-host allowlist, and a written rubric for the LLM judge so “acted on a poisoned fact” is a checkable claim. The second problem was silent failure in the plumbing. Sonnet 5.5’s extended thinking consumed the whole token budget and returned empty final answers, which a naive scorer would have counted as “held”, so we record stop_reason and treat an empty answer as a harness error, not a pass. SurrealDB Agent Memory takes scopes on writes but lens on reads and ignores unknown fields, so a read with the wrong key quietly searches everything the key can see; we caught it because the poisoned fact showed up where it should not have. And the Python 3.13 build on our machine shipped without a CA bundle, so every Daytona call failed TLS until we pointed it at certifi. None of these are glamorous. All of them are exactly the kind of quiet wrongness a red-team harness exists to catch, which is why we built the harness to distrust itself first.

Accomplishments that we’re proud of

The baseline is fair: round 1 is not sabotaged, and round 2’s rules came from round-1 evidence plus recalled memory, not from us. Real numbers from today, all on real sandboxes with tracing on: Claude Sonnet 5.5 held 7 of 8 undefended, the one that landed was the rm -rf hidden in a cleanup guide, and the memory-hardened round 2 held all 8. Claude 3 Haiku held 8 of 8 in both rounds. We reported that as found instead of tuning attacks until the board turned red.

What we learned

Provenance beats filtering. The agent never had to detect the poisoned policy; it only had to see which scope the fact came from. Every fact we write carries a scope and a timestamp, and the round-2 recall traces back to the exact round-1 results that produced it. Memory with sources is a defense, not a feature.

What’s next for Crashtest

Serve the hostile pages from a Daytona preview URL so Anchor visits a genuinely compromised site. Turn the hardener into an Arize Prompt Learning loop. Add Bedrock Guardrails as a second arm and report attack success with and without them. Grow the attack library from eight hand-written scenarios to generated variants.