-
-
Same 16-case corpus: 16 critical failures in vulnerable, 0 in fixed.
Inspiration
I originally started LLMFuzz because I wanted to build a real developer tool that could make meaningful use of my local AI hardware and help me learn through a practical project.
When OpenAI Build Week was announced, it gave me a concrete reason and deadline to extend LLMFuzz beyond traditional byte-level fuzzing. AI agents can continue running normally while still making serious semantic mistakes, such as following injected instructions, revealing protected information, using forbidden tools, or bypassing approvals.
What it does
LLMFuzz Red Team uses GPT-5.6 through the OpenAI Responses API to generate a bounded, structured corpus of realistic attack scenarios for AI agents.
The current corpus covers four risk classes:
- prompt injection
- secret exfiltration
- forbidden tool use
- approval bypass
The generated corpus is strictly validated, canonically persisted, identified by a reproducible hash, and reusable without another model call.
The same corpus is executed against a deliberately vulnerable target and a minimally fixed version of that target. Both targets emit machine-readable events. Deterministic invariants—not an LLM judge—then calculate the final PASS, FAIL, TIMEOUT, or BLOCKED verdicts.
How I built it
I built LLMFuzz Red Team using an AI-native development process. I worked with ChatGPT on the product direction, architecture, specifications, and implementation plans, and used Codex for implementation, testing, and separate review and audit passes.
I divided the work into bounded phases and kept implementation separate from verification. Codex was used both to create the code and to inspect the resulting changes through focused test, review, and audit workflows.
GPT-5.6 was used through the API for the part where generative capability is most useful: creating varied adversarial test cases. It was intentionally not used to decide whether a target passed or failed. Final verdicts come only from deterministic rules applied to machine-readable evidence.
Challenges I ran into
Phase C, which covered corpus generation, validation, persistence, and API integration, took longer than the deterministic execution and reporting work in Phase D.
The difficult part was not simply getting a model to generate attacks. It was turning generated output into bounded, validated, reproducible test input that could later be replayed offline and judged without relying on another model call.
Another challenge was maintaining a disciplined separation between implementation, review, testing, evidence collection, packaging, and the final submission process.
What I learned
The project made the weaknesses of AI agents much more concrete to me. An agent does not need to crash to fail. It can remain operational while following an untrusted instruction, exposing protected data, calling a forbidden tool, or performing an action without valid approval.
I also learned that useful AI security testing requires more than persuasive output from another model. Results need machine-readable evidence, fixed evaluation rules, and enough determinism to be repeated during regression testing and CI.
What's next
The next step is to make LLMFuzz Red Team easier to connect to real AI-agent pipelines and assistants.
A possible future direction is a hosted API or remote testing service running on my own infrastructure, while preserving the ability to replay corpora and verify results deterministically and offline.
Log in or sign up for Devpost to join the conversation.