• 

  demoscreen

Agentic Adversarial Attacks on PII Detection Systems

Inspiration

There is a vacuous space between what an AI system is supposed to do and what happens in the real world. For so long, I have sought to fill this void by answering the question: how do we turn ethical principles—responsible AI, fairness, privacy, and human-centered design—into systems that can actually be inspected and tested?

PII detectors often make use of LLM-as-a-judge detection systems. Such a classifier may correctly recognize an email address, telephone number, or account identifier when it appears in a familiar sentence. But real language rarely stays familiar. People abbreviate, reformat, paraphrase, encode, split, disguise, and rearrange information.

This inspired my red-teaming tool for testing the robustness of PII classifiers: an interactive developer tool that reveals what happens when several cooperative AI agents repeatedly transform sensitive-looking text, test a PII detector, and share the strategies that work.

The simulation was inspired by the wonderfully strange stealth game Sneak King. Instead of sneaking up on unsuspecting characters with burgers, these agents sneak transformed inputs toward a classifier’s decision boundary. Each agent has its own tactics. The most successful strategy is passed around the swarm, and the strongest agent ultimately earns the crown.

Behind the playful presentation is a serious goal: make privacy-classifier weaknesses visible before they become real-world failures.

What it does

This tool allows developers and safety engineers to observe an entire cooperative mutation process rather than receiving only a final score.

The system:

• Configures four or six mutation agents.
• Assigns one or more transformation strategies to each agent.
• Begins with known-positive PII examples.
• Records the detector’s expected baseline response.
• Applies mutations over multiple rounds.
• Shows the detector verdict after every attempt.
• Preserves both accepted and blocked mutation paths.
• Tracks which agent created each transformation.
• Shares the strongest successful strategy at round boundaries.
• Ranks agents according to their results.
• Compares baseline detection with post-mutation outcomes.

A useful way to express the change caused by a mutation is:

Δc = c_baseline − c_mutated

Here, c_baseline is the detector’s original confidence and c_mutated is its confidence after a transformation.

A large positive value of Δc indicates that the detector became substantially less confident even though the important identifier remained present.

The purpose is not simply to produce adversarial text. It is to expose the path that led there: which strategy was attempted, why it passed or failed, what another agent inherited, and how the detector’s behavior changed over time.

How I built it

The project has two connected layers.

Interactive browser layer

The browser interface is a standalone HTML application that presents the mutation process as an animated agent observatory.

It includes:

• Agent configuration and strategy selection
• Mutation rounds
• Explainable detector verdicts
• Accepted and blocked transformation paths
• Strategy inheritance between agents
• A Skip to Finish system that completes remaining attempts without erasing finished work
• Final agent rankings and performance comparisons

The browser layer uses a deterministic teaching simulator so the experience is repeatable and can be explored without downloading machine-learning models.

Classifier-backed Python layer

The Python backend provides the path to real model evaluation.

It uses:

• Roblox/roblox-pii-classifier
• Roblox/RobloxGuard-Eval
• FLAN-T5 for text transformations
• Sentence embeddings for semantic-similarity checks
• Reproducible seeded sampling
• Labeled classifier evaluation
• Precision, recall, specificity, F1, balanced accuracy, and confusion counts

The mutation pipeline checks whether a transformed sentence remains sufficiently close in meaning to its source. A conceptual acceptance condition is:

sim(x, x′) ≥ τ

Here, x is the original input, x′ is the mutation, and τ is the minimum semantic-similarity threshold.

This prevents the system from treating a meaningless rewrite as a successful mutation simply because the classifier score changed.

Codex and GPT-5.6

Codex served as my primary engineering collaborator.

I used it to inspect the inherited prototype, trace state across a large single-file interface, diagnose incorrect strategy attribution, redesign agent configuration, preserve mutation lineage, implement Skip to Finish, add final analysis, and strengthen the project structure.

GPT-5.6 was used to generate a set of semantic mutations while preserving the identifiers required for classifier testing. Those outputs were then evaluated using the real Roblox classifier and stored as a reproducible result artifact.

Codex was most valuable not as an autocomplete tool, but as a collaborator capable of reasoning across the interface, experiment pipeline, state model, evaluation outputs, and presentation layer together.

Challenges I faced

Preserving meaning without preserving phrasing

The hardest conceptual problem was deciding what counts as a valid mutation.

Changing a classifier score is easy when the text’s meaning is destroyed. That result is not useful. The transformed text needed to preserve the relevant identifier and broadly retain the original intent while changing its surface form.

I addressed this through identifier-preservation checks and semantic-similarity filtering.

Separating simulation from measured results

The browser experience needed to be immediate, visual, and deterministic. The real classifier backend required model downloads and could run slowly on a CPU.

Combining the two without clearly distinguishing them could produce misleading claims.

I therefore kept their responsibilities explicit:

• The browser interface teaches and visualizes the process.
• The Python backend performs real classifier evaluation.
• Dataset-wide metrics are only claimed when a labeled evaluation has actually run.

Tracking mutation lineage

Once agents began sharing strategies, attribution became difficult.

A mutation might be generated by one agent using a strategy inherited from another agent, based on an input mutated in an earlier round. Showing only the final string would hide the most interesting part of the experiment.

The project therefore preserves the complete lineage: parent input, originating agent, current agent, attempted strategy, verdict, round, and resulting score.

Working under a deadline

The final challenge was scope.

There were many directions the project could have taken: more models, larger datasets, additional mutation families, richer charts, or a more complex backend. The difficult decision was identifying what best communicated the central idea. For this purposes, the mutations and starting PII samples were chosen to show a complete end-to-end process that I am proud of delivering in full.

I also learned that agent cooperation offers new avenues to red-teaming exercises. Cooperative agents create an evolving search process in which tactics spread and combine, allowing unique ways to discover flaws in PII detection systems.

That makes observability essential. As agent systems become more capable, safety tooling must show not only what an agent produced, but how information and strategies moved through the system. The crown does not simply go to the agent that produces the strangest text. It goes to the agent that most effectively reveals a meaningful weakness while preserving the information being tested.

What is next

The long-term goal is to create a tool that helps teams move from saying a detector is “accurate” to understanding precisely where it is reliable, how it fails, and what should be improved.

Until then, the agents will keep moving through the shadows—testing the guards, exchanging tactics, and competing for the crown.

Built With

• chatgpt
• codex
• cybersecurity
• gpt5.6
• javascript
• pii
• python
• red-teaming