Swarm Chaser

Inspiration

In November 2025, Anthropic reported the first documented cyber-espionage campaign largely run by an AI agent. Agents now read files, send email and run commands for us, and they can be steered into causing harm. We wanted to see whether machine learning could catch a malicious agent before it acts.

What it does

Swarm Chaser monitors AI agents from two sides. A black-box model scores each agent's visible behaviour (its messages, tool calls and interactions) for signs of malicious activity. A white-box model reads the agent's internal activations at every decision and predicts whether it is acting with malicious intent, before the action happens.

How we built it

• Sandbox environment: a Python agent harness that gives open-weight models (Qwen2.5-3B, Gemma-2-9B, Gemma 1B) a set of simulated tools that look real but take no real action, so agents can be given harmful tasks safely.
• Experimental dataset collection: we labelled benign and malicious tasks; at every agent step we capture the model's hidden-state activations with PyTorch and Hugging Face Transformers.
• White-box model: a logistic-regression probe (scikit-learn) trained on our own activation dataset, one per agent model, validated on attack types held out from training.
• Black-box model: a classifier trained on the MRT agent-trajectory dataset.
• Platform: Tiger Data for storage and a web front end for monitoring.

Challenges we ran into

Cloud container deployment, the front-end user experience, getting safety-tuned models to behave like real agents in a sandbox, and sleep deprivation.

Accomplishments that we're proud of

• Built our own experimental datasets of agent activations captured during inference, and trained probes that detect malicious intent on attack types they never saw in training.
• Built a working end-to-end pipeline, from sandbox to dataset to trained detector, in under 14 hours.

What we learned

• Small open-weight models are highly compliant; Qwen 2.5-3B carried out every malicious task we gave it as an agent, with no jailbreaking required.
• A model's internal activation state carries lots of signal about the intent of the agent, that may be invisible to behaviour-only monitoring frameworks.

What's next for Swarm Chaser

Testing on independent benchmarks such as AgentHarm, adding more models, measuring cases where agents refuse, and running the white-box monitor live alongside deployed agents.

Built With

• docker
• joblib
• pandas
• python
• pytorch
• react
• scikit-learn
• tigerdata
• typescript
• vercel