tripwire

Inspiration

AI assistants that read email are now common, and an email is text written by someone else. An attacker can hide an instruction inside it, and the assistant may obey that instead of its owner. Checking for this by hand is slow and subjective, so I wanted to automate it: attack an assistant, score the result with fixed rules, add defences, and measure what changes.

What it does

Tripwire automates red-teaming of an AI email assistant.

• A small assistant (Qwen3-32B via Featherless) reads an email and returns a summary, tasks and a draft reply. It holds a fake secret, the canary.
• A library of 20 hand-written attack emails (5 techniques, 4 each: override, fake system notice, hidden text, role-play, exfiltration) plus 6 normal control emails is run against it.
• A deterministic scorer decides whether each attack worked. No model grades the model.
• Five toggleable defences (delimiters, strip_hidden, flag_phrases, output_filter, schema) can be applied alone or together.
• Runs are cached and recorded, so every result can be replayed without an API key. A --live switch asks before making any API call.
• Two front ends read only the recorded results: a Streamlit app (overview, walkthrough, heatmap, results, an attack inspector) and a static replay site in plain HTML, CSS and JavaScript that works offline.

How I built it

Python, Streamlit, and Qwen3-32B served by Featherless. The scorer counts an attack as a strict success if the canary leaks, or if the model produces the attacker's marker as its own output. A looser mentions measure counts the marker or canary appearing anywhere, even inside a refusal. Control emails must still get a useful reply, so a defence cannot score well just by refusing everything.

The runner has a reply cache, a call cap, a timeout and a confirmation prompt before any live call, and failed calls are skipped and reported, never counted as blocked. Both front ends escape all email and reply text, and tests scan the code for unsafe HTML and external requests.

Results

All data is fictional, with one model.

A repeat baseline run scored 14 of 20 (70%), so the baseline itself moves by about 7 points. The schema-only and strip-hidden runs fall within that spread, so only the large gaps are meaningful. The best single defence (output filter) was 25%, and all five together reached 10%.

Challenges and what I learned

• My first scorer overcounted. It treated refusals that quoted the attacker's text as successful attacks. I found this by reading the replies, tightened the rules and re-scored the baseline. The README says so.
• The model provider was unreliable under load. Some calls returned errors or were cut off. Failed calls are reported and left out, never scored as blocked. A third baseline repeat was incomplete for this reason and is excluded, and a second was discarded after a failed attempt to resume it.
• Some successes are arguable. In my own judgement, the baseline has 8 to 12 of 19 real successes and the defended run 1 to 2 of 20, depending on how borderline replies are counted.
• Measuring honestly mattered more than a dramatic number: report the spread, show the failures, and say what the test cannot show.

Limits

20 hand-written attacks (one attack moves a rate by 25 points), one model, mostly single runs, fictional data. This measures one assistant against these attacks. It does not show that any real assistant is safe.

AI use

Claude (chat) helped plan and review. I reviewed the diffs and ran the tests myself. Ideas such as treating message text as untrusted and checking model output came from an earlier project, but no code was copied. Built from 10 October 2026, after kickoff.

What's next

More attacks and models, repeated runs of every configuration, and testing each defence's effect more carefully.

Built With

• claude
• css
• featherless
• html5
• javascript
• pytest
• python
• qwen
• streamlit