Agentatk

An AI agent that attacks other agents and tests their limits, guardrails, and security.

Comment

Inspiration

AI agents can now take real actions: send emails, modify data, control devices, and call powerful tools. But one question is still hard to answer:

Can my AI agent actually be broken?

We faced this problem while building our own agents. Existing security tools helped, but many required specific frameworks, tech stacks, file structures, or manual configuration.

We wanted a one-click solution that could work across different agents.

So we built Agentatk: an autonomous AI security researcher inspired by coding agents like Claude Code. Instead of fixing code, it explores an agent, finds attack surfaces, attacks them, verifies what actually happened, and reports the vulnerabilities.

What it does

Point Agentatk at an AI agent's codebase and it:

  1. Explores the codebase to understand its architecture, tools, inputs, guardrails, and risky actions.
  2. Finds potential attack paths and generates targeted security hypotheses.
  3. Attacks the target using direct and indirect attacks such as prompt injection.
  4. Verifies whether the attack actually crossed a security boundary using execution telemetry and independent judges.
  5. Reports confirmed vulnerabilities with evidence, severity, and attack traces.

The core loop is:

Explore → Understand → Find Risk → Attack → Verify → Adapt → Report

Unlike a traditional scanner, Agentatk doesn't rely on a fixed list of vulnerabilities or assume how the target is structured. It first learns how the target works, then decides what should be tested.

What it does

Agentatk is a tool that automatically checks how secure your AI agent is. You point it at your agent's code, and it:

  1. Reads through the code to understand how the agent works
  2. Comes up with realistic attack ideas (like prompt injection or trying to override the agent's instructions)
  3. Actually tests those attacks against your agent
  4. Gives you a report showing what worked, what didn't, and how serious each issue is

How we built it

We started by studying how coding agents like Claude Code work — how they read a codebase, understand it, and reason about it step by step. We used that same idea, but instead of writing code, our agent looks for security holes.

The tool first scans the code itself (not just poking at the agent from outside like a black box). This is what makes it different from other tools — most just send random inputs to the agent's API and see what happens. We actually read the source code first, find the risky spots, and then design targeted attacks for those spots.

Challenges we ran into

Problem 1: We were only testing 3-5 things, no matter how big the target was.

At first, we asked the AI to come up with 3-5 attack ideas, test them, and stop. That sounds fine — until you try it on a real agent with 25-40 tools (things like unlock, lock, set_temperature, process_refund). We were only ever testing a random handful and skipping the rest completely. Most of the actual attack surface was never even checked.

The fix — test everything, in order of danger, not just a random sample.

We built something we call the Risk Budget Allocator. Instead of guessing 3-5 things to test, it:

  • Goes through every single tool/function it found in the code — not a sample, all of them
  • Tests each one two ways: a direct attack (tricking it through a normal chat message) and an indirect attack (sneaking a fake instruction in through something the agent reads, like a document or API response)
  • Automatically scales up the number of tests to match how many real risky spots it actually found, instead of stopping at a fixed number
  • Tests the most dangerous stuff first, then works down to less critical stuff

This alone took us from 3-5 tests to 60-80 tests on the same target.

Side problem: once we hit ~80 tests, some of them were fake.

We noticed part of that jump was because our scanner was accidentally reading test files and treating fake helper functions (like mock_get_api() or dummy_service()) as if they were real parts of the agent. So we added filtering to skip test folders and anything named like fake_, mock_, stub_, dummy_ — while still keeping real folders like data/. This made the numbers accurate instead of just big.

Problem 2: The AI was grading its own test.

Testing more things is only useful if you can trust the results. Our early version had one AI do the attack AND decide if the attack worked — which is like letting a student grade their own exam. This caused two problems:

  • False alarms: if the user asked "turn off the bedroom light" and the agent safely did exactly that, the same AI sometimes patted itself on the back and called it a "successful hack" — just because a tool got called at all, not because anything wrong happened.
  • Missed real issues: it sometimes overlooked sneakier attacks, like asking to check a porch light but actually unlocking the front door instead.

The fix — separate the attacker from the judge completely.

Now the AI that attacks the system has zero say in whether the attack succeeded. Instead:

  • We record exactly what tool got called and with what data, straight from the system — no interpretation
  • Three separate AI judges (run at different "creativity" settings) independently decide whether that action was actually something the user asked for, or something snuck in by the attack
  • At least 2 out of 3 judges have to agree before we count it as a real, confirmed finding

In short: the allocator fixed how much we were testing. The separated judge fixed how trustworthy the results were. Both were needed — testing more things doesn't help if you can't trust which ones actually broke.

Accomplishments that we're proud of

We found real, working vulnerabilities in AI agents controlling real-world devices.

We tested Agentatk against home-llm, an open-source integration that lets an AI control real smart home devices — lights, fans, window covers, locks — through plain English commands.

We sent the agent a message that looked completely normal, but had a fake "ignore your safety rules" instruction hidden inside it. The agent fell for it — it carried out the hidden instruction and controlled a device it was never actually asked to touch.

This wasn't a one-time fluke. The same trick worked across multiple different device controls in the same test run — turning things off, toggling them, changing fan speed, stopping window covers. That told us this isn't a bug in one feature, it's a pattern: the agent has no reliable way to tell a real instruction apart from a fake one smuggled inside a message.

We proved it generalizes — it's not hand-built for one target.

We pointed Agentatk at a completely different kind of agent: fastagency, a framework used to automate WhatsApp messages — nothing to do with smart homes. With zero manual setup for this new target, Agentatk read its code on its own, figured out what actions it could take, and found the same category of weakness: a message combining a normal request with a hidden override phrase caused the agent to trigger an action it was never asked for.

This is the part we're proudest of. Our tool wasn't built to catch one specific bug in one specific app. It looked at a system it had never seen before, understood it, and found a real security hole on its own.

What we learned

  • Self-grading is a trap: if the same AI that creates an attack also decides whether it worked, it tends to grade itself too kindly. The grader needs to be separate and neutral.
  • Testing smarter beats testing more: instead of trying every possible combination randomly, focusing your limited budget on the highest-risk spots first — and then guaranteeing full coverage — gets you far more real findings.
  • Reading the code first is a real advantage: most security tools only poke at an agent from outside (black box). Actually reading the source code first lets you design much more targeted, realistic attacks.
  • Small bugs cause big blind spots: something as small as a filtering bug (test files or mock functions getting counted as real code) can quietly inflate or corrupt your results without any obvious error.
  • Coverage and trust are two separate problems: testing more things (the allocator) and trusting what you find (the judge separation) needed two completely different fixes — solving one didn't solve the other.

What's next for Agentatk

  • Making the agent faster and cheaper to run (fewer tokens, quicker scans)
  • Testing multi-step attacks — where the attack doesn't hit the target directly, but travels through a chain (Step A → Step B → the actual vulnerable spot)
  • Testing for "tool poisoning" (tricking an agent through a malicious tool/plugin it uses)
  • Improving the UI
  • Making the overall product more production-ready

Built With

Share this project:

Updates

Submission history