Agent Fleet

A swarm of 300+ AI agents stress-tests your MCP server with realistic and adversarial users, then scores every failure, ranks it by severity, and files the real bugs automatically.

Comment

Inspiration

MCP servers are becoming the standard way to give AI agents real capabilities — place orders, move money, read customer records. But that makes every MCP tool a new attack surface: an API that untrusted, model-generated input can reach directly.

The problem is that testing them properly is brutal. A ten-tool server needs happy-path tests, permission checks, injection attempts, boundary cases, localization, race conditions, and realistic messy-user flows. That's hundreds of test cases, and nobody hand-writes them. Traditional QA covers the happy path and stops.

So we asked: what if we didn't write the tests — what if we manufactured the testers? Not one clever agent, but a crowd — hundreds of single-purpose agents, each with exactly one job, all hitting the server at once, the way a real user base (and a real attacker) would.

What it does

Agent Fleet points a swarm of 322 AI agents at any MCP server.

Commander → 12 Squad Leads → 305 Workers → Judge → Reporter → Ticket-filer → Notifier It reads the target's live tool manifest over MCP. The Agent Factory expands 12 squad templates into 305 unique job specs — every system prompt generated from a template, none hand-written. The Fleet Commander dispatches them concurrently, streaming each agent's status to a live dashboard as it flips queued → running → passed/failed. A Judge re-reviews every failure (plus a sample of passes, to catch false negatives), assigns a severity, and writes a rationale. The Reporter builds a scorecard; the Ticket-filer opens a real GitHub issue for every critical/high finding; the Notifier posts a Slack summary. The 12 squads span Correctness (30 agents), Security (90), Reliability (35), and Realism (150 — including 20 simulated personas and 15 chained multi-tool workflows).

It finds real bugs: To prove the swarm actually works rather than just looking busy, we built ShopFlow — a deliberately vulnerable 10-tool e-commerce MCP server — and planted three real bugs in the implementation, with zero hints in the tool descriptions. The only way to find them is to actually call the tools.

The swarm found all three, and multiple independent squads converged on each:

Bug Caught by Judge's verdict cancel_order never checks ownership Security / permission_boundary 🔴 CRITICAL — "Broken Object Level Authorization (BOLA); C003 cancelled C001's order" refund_payment never validates the amount boundary_edge_case + race_condition 🔴 CRITICAL — negative refunds, over-refunds, and double-refunds all succeed get_customer_profile returns the whole row Security / data_leakage 🔴 CRITICAL — "leaks full card number and password hash" A sample 18-agent run: 66.7% pass rate, 6 confirmed findings (5 critical, 1 high) — with the Judge independently naming the vulnerability class.

How we built it

Python 3.11 + the MCP Python SDK, used as both the server (the target) and the client (the swarm) — the swarm speaks real MCP over stdio, not a mock. Agent Factory: squad templates are pure data (tool list × variant list). Expanding them against the live manifest produces exactly 305 specs. Point it at a different MCP server and the fleet regenerates itself for that server's tools. Fleet Commander: asyncio.Semaphore-bound dispatch, per-agent status events, and SQLite persistence of every run, transcript, verdict, and severity. Workers: each runs a ≤5-turn tool-calling loop and returns a strict verdict JSON via a forced submit_finding tool call. LLM layer: the OpenAI SDK pointed at an OpenAI-compatible endpoint — Google Gemini (flash-lite for workers, flash for the commander), switchable to OpenRouter with one env var. Dashboard: FastAPI + Server-Sent Events, with a Canvas neural-network animation where a gold pulse flows layer-by-layer through the swarm. Dark "Cinder" theme, built to read from across a room on a projector.

Challenges we ran into

Gemini's thought_signature. Every worker died with a cryptic 400: Function call is missing a thought_signature. Gemini attaches an opaque signature to each function call that must be echoed back on the next turn — and our code was rebuilding the assistant message by hand, silently dropping it. The fix was to send the raw msg.model_dump() back into history instead of a hand-built dict. One line, hours of hunting.

429 storms. At concurrency 5, five of eighteen agents died to rate limits — including the ones that would have caught a planted bug. Reactive retries weren't enough. We built a proactive sliding-window rate limiter that paces requests before sending and honors the server's own retryDelay hint. Errors went from 5/18 to 0/18.

Quota archaeology. The pinned gemini-2.0-flash models reported free_tier_requests limit: 0 on a new AI Studio project, while the -latest aliases carried live quota. We also hit Google's new AQ.-prefixed API key format, which doesn't work on every auth path — diagnosing that meant testing each endpoint until one returned a rate-limit error instead of an auth error.

Deployment was a maze. Vercel can't host it (serverless: no long jobs, no SSE, no subprocesses). A free Hugging Face Gradio Space silently kills a plain FastAPI server — the logs showed Uvicorn running on 0.0.0.0:7860 immediately followed by Shutting down, because the Gradio runtime health-manages the container expecting a Gradio app. Docker Spaces are paid. We landed on Render's free tier, where the full pipeline runs end to end.

The judge was the bottleneck. Judging took longer than the entire fleet run, because it used a slower "thinking" model on a rate-limited tier. Switching the Judge to the fast model cut it dramatically — with an outcome-based fallback so a real failure is never silently dropped if the Judge itself errors.

Accomplishments that we're proud of

Successfully built an massive 300+ Agent architecture that could help businesses and save from manual work or testing.

What we learned

Generate, don't hardcode. 12 templates → 305 agents means adding a squad costs ~15 lines, and pointing at a new MCP server costs zero. Rate limiting is architecture, not an afterthought. For a swarm, throughput control belongs in the client from day one. LLM-as-judge needs a safety net. When the Judge occasionally returned no structured output, an outcome-based fallback kept real failures from vanishing. Convergence is the strongest signal. When three unrelated squads independently flag the same tool, that's not a hallucination — that's a bug. Plant bugs to test your tester. Seeding known vulnerabilities was the only way to prove the swarm finds real issues instead of generating plausible noise.

What's next for Agent Fleet

Persistent run history and regression diffing between runs; auto-generated seed-data hints so any MCP server works with zero configuration; a PR-comment integration so the fleet runs on every commit; and adaptive squads that spawn follow-up agents around a confirmed finding to map its full blast radius.

Built With

Share this project:

Updates