TripWire

Tripwire attacks your chatbot like a real hacker, catches leaks with canary tokens, and fixes them with a hardened prompt, a self-trained ML firewall and an output guard.

Comment

Inspiration

Everyone is launching chatbots: cafés, universities, HR teams. Most of them are just a system prompt behind a chat box, and almost nobody checks whether they can be tricked. One cleverly worded message can make a bot reveal its hidden instructions, leak a confidential code, or break its own business rules. I don't come from a security background, so I wanted to build something that lets anyone, including people like me, test their chatbot the way an attacker would, before it goes live.

What it does

Tripwire is an automated red team for chatbots. You pick a sandbox bot (or paste your own system prompt, rules and the secret you want to protect), and Tripwire:

  • Attacks: fires 30 real techniques across 6 categories: prompt leaks, instruction overrides, role-play jailbreaks, indirect injection through documents, encoding tricks and policy violations.
  • Detects: plants a hidden canary token in the prompt and checks every reply for it, even when disguised (spaced out, reversed, base64). An LLM judge scores subtler rule-breaking, and disagreements are flagged instead of guessed.
  • Explains: gives a security score and grade, maps results to the OWASP Top 10 for LLM Applications, and gives prioritised recommendations. For example, "Remove secrets from the system prompt entirely".
  • Defends & re-tests: applies three layers (an AI-hardened prompt, a prompt firewall I trained myself, and an output guard), then re-runs the same attacks so you can see the before and after.
  • Checks usability: runs normal customer questions too, so a bot that is "secure" because it refuses everything doesn't get a free pass.

On my IT helpdesk test bot, the baseline scored 93% (grade A), yet it still leaked its confidential code twice through encoding tricks. A high score can hide a real leak.

How I built it

  • Python + Streamlit for the app, deployed on Streamlit Community Cloud.
  • An LLM API (OpenAI-compatible, Gemini) for four roles: the simulated target bot, the judge, the prompt hardener and the recommendation writer.
  • scikit-learn for the firewall: TF-IDF (word + character n-grams) with a linear classifier, trained on the public deepset/prompt-injections dataset plus my own attack and normal examples, and evaluated on held-out data (precision, recall, confusion matrix, threshold trade-offs).
  • Deterministic checks (canary detection, output guard) as ground truth, so the most important verdicts don't depend on an LLM.
  • 100+ automated tests, a mock mode and scenario test bots built to fail in specific ways.

The security score counts each tested attack:

$$\text{Score} = \frac{\text{resisted} + \text{blocked} + 0.5 \times \text{partial}}{\text{attacks tested}}$$

Challenges I ran into

  • API rate limits: the free tier kept returning 429 errors, and half my scans came back empty. I cut the number of calls by batching the judge, caching results, skipping the judge when the canary already proved a leak, and adding a resumable "not tested" state.
  • The "perfectly secure" bot: a bot that refuses everything scored an A. That's when I added the normal-question check, because security that breaks the product isn't really security.
  • Trusting the judge: LLM judges can be wrong, so I added double-judging and made the deterministic checks override it.
  • Environment issues: scikit-learn wouldn't install on Windows with a very new Python until I pinned compatible versions, and I also had to fight Streamlit's layout to get a clean full-width design.

Accomplishments that I'm proud of

  • Watching the canary light up red the first time a bot leaked its secret.
  • Training and evaluating my own ML model and using it as a real defence layer, not just an LLM wrapper.
  • Building something a non-expert can actually use, with honest limitations instead of overclaiming.

What I learned

  • How prompt injection and jailbreaks work, and how they map to the OWASP Top 10 for LLMs.
  • Proper ML evaluation: held-out test sets, precision vs recall, and choosing a threshold that balances security against usability.
  • LLM-as-a-judge is useful but not trustworthy on its own. Deterministic checks are essential.
  • How to direct AI coding tools: knowing what to ask for, and checking what comes back.

What's next for TripWire

  • Calibrating the firewall to each business's own normal traffic, and letting users retrain it on their own examples.
  • Multi-turn and non-English attacks.
  • Deeper testing for tool-using agents and multi-agent pipelines.
  • Safely testing live chatbot endpoints, with checks that the user owns them.

Built With

Share this project:

Updates

Submission history