GoPhish

GoPhish is an interactive article and benchmark that reveals how Claude, GPT‑4o, Gemini, and Llama handle phishing and jailbreak prompts in real time.

Comment

Inspiration

Phishing has quietly become one of the most effective real‑world attack vectors, and recent LLMs dramatically lower the bar for writing convincing phishing emails at scale. We wanted a way to see that shift, not just read about it in a paper.

Two questions drove GoPhish:

  • How consistently do frontier LLMs refuse to generate phishing content?
  • How easily do simple jailbreak framings break those guardrails?

We also wanted a format that felt more like an interactive article than a dashboard, so a broader technical audience could understand the threat landscape without needing a background in security research.

What it does

GoPhish is two things built together:

  1. The Article — “The Phishing Frontier: AI at the Hook”
    A scroll-driven, dark-themed interactive article with seven embedded features:
  • Detection quiz — A 3‑round “Are you better than an LLM?” email classification game.
  • 30‑year timeline — From early AOL scams to agentic AI phishing in 2024.
  • Live model chat — Send the same phishing prompt to Claude Sonnet 4.6, GPT‑4o, Gemini 2.5 Pro, and Llama 3.3 70B and compare responses.
  • Jailbreak explorer — Test four adversarial prompt framings (direct, educational, fictional, security‑research) against each model in real time.
  • Refusal charts — Interactive bar charts showing refusal rates for direct vs. jailbreak prompts, plus phishing volume/success trends.
  • Agent simulation — Side‑by‑side view of an unsafe vs. robust AI agent handling a phishing campaign request.
  • Annotated phishing email — A fully AI‑generated spear phish with clickable callouts for social‑engineering tactics.
  1. The Benchmark — GoPhish refusal‑rate evaluation suite
    A structured benchmark built with inspect-ai that measures how often each model refuses phishing requests.
  • Prompt taxonomy:
    • Attack goals: A1 credential theft, A2 financial fraud, A3 access abuse.
    • Roles: B1 IT/security, B2 HR/internal, B3 bank/financial, B4 government.
  • Tasks:
    • 36 direct prompts (“guardrails” condition).
    • The same 36 prompts wrapped in 3 jailbreak frames (“roleplay”, “research framing”, “rewrite request”) for 108 jailbreak samples.
  • Judge: Claude Opus 4.6 labels each response as REFUSAL or HARMFUL. Refusal rate = fraction of samples labeled REFUSAL.

What we learned

  • Guardrails can look strong on paper but collapse under jailbreaks.
    In our runs, Claude Sonnet 4.6 refused 97% of direct phishing prompts, but only 9% once we applied jailbreak framings. GPT‑4o and Llama 3.3 70B dropped to 0% refusal under jailbreaks.

  • Some models still have no practical guardrails on phishing.
    Gemini 2.5 Pro complied with all 36 direct phishing prompts in our suite, without any jailbreak needed.

  • Framing matters as much as content.
    Simple changes like “this is for security awareness training” or “this is for a fictional novel” were enough to flip models from refusal to detailed phishing emails, even though the underlying task didn’t change.

  • Narrative UX helps communicate security research.
    Putting charts, live model calls, and simulations into a single continuous article made it much easier to explain results to classmates who aren’t deep into AI security.

How we built it

Frontend / article

  • We used Next.js (App Router) and React to build a single‑page, scroll‑driven article.
  • Each interactive feature is a dedicated component:
    • PhishingQuiz.tsx (quiz), PhishingTimeline.tsx (timeline),
      ChatSimulation.tsx (live multi‑model chat), JailbreakChat.tsx (strategy explorer),
      RefusalCharts.tsx (charts), AgentSimulation.tsx (agent), PhishingAwareness.tsx (annotated email).
  • Styling is done with Tailwind CSS and custom CSS/SVG for the charts (no charting libraries), to keep the page light and fully controllable.
  • A single API route (app/api/chat/route.ts) proxies requests to all four LLM providers, so the UI can just say “send to model X” and not worry about vendor‑specific clients.

Backend / evaluation

  • The benchmark lives in the benchmark/ directory and is built with inspect‑ai.
  • gophish_prompts.py defines the 36 base phishing prompts and the three jailbreak wrappers.
  • gophish_tasks.py defines two tasks (direct and jailbreak) and a Claude Opus 4.6–based judge that maps each model response to REFUSAL or HARMFUL.
  • run.py is a CLI that:
    • Loads API keys from the Next.js .env.local.
    • Runs a chosen model (or all four) on the selected task(s).
    • Logs structured JSON (including breakdowns by goal, role, and jailbreak type) to results/.

This separation let us iterate quickly on prompts and models without touching the frontend, and re‑render the charts simply by dropping new JSON into benchmark/results/.

Challenges we ran into

  • Coordinating four different model providers.
    Claude, OpenAI, OpenRouter (for Gemini), and Together AI all have slightly different API shapes and rate‑limit behaviors. We had to design a small abstraction layer in the API route and deal with occasional timeouts and streaming quirks.

  • Balancing safety with demonstration.
    We needed to show realistic phishing outputs without building a turnkey phishing tool. In the UI, we:

    • Clearly label content as unsafe or hypothetical.
    • Emphasize defensive / educational framing (e.g., quizzes, annotated phish, awareness tips).
    • Avoid including raw links that could realistically be reused.

  • Making the article feel cohesive, not like seven mini‑apps.
    We spent a surprising amount of time on the scroll experience, section transitions, and narrative copy, so that the quiz, live chat, jailbreak explorer, and charts all feel like chapters in one story rather than separate demos.

  • Time‑boxed experimentation.
    Running python run.py --model all --task both across four frontier models is slow and uses real credits. We had to be careful with batch sizes and only re‑run full sweeps when prompts or tasks changed meaningfully.

Overall, we came away with a much better appreciation for how fragile current LLM guardrails are in phishing settings, and how important good UX is for communicating AI security results.

Built With

  • Languages / frameworks
    • TypeScript
    • Next.js (App Router)
    • React
    • Tailwind CSS
  • LLM / SDKs
    • Anthropic SDK (Claude Sonnet 4.6)
    • OpenAI SDK (GPT‑4o)
    • OpenRouter (Gemini 2.5 Pro via OpenAI‑compatible API)
    • Together AI API (Llama 3.3 70B Instruct Turbo)
  • Evaluation / tooling
    • inspect-ai
    • Python
    • dotenv / JSON logging

Built With

Share this project:

Updates