DeepAudit.ai

Inspiration

AI chatbots are shipping faster than ever, often built by small teams racing to launch. Many of these bots carry sensitive business logic directly in their system prompts, pricing rules, internal policies, escalation procedures, under the assumption that users will simply never think to ask for it. But prompt injection and jailbreak techniques are well-documented and easy to attempt, and most small teams don't have the time or security expertise to systematically test for this before launch.

I wanted to build the tool I wished existed: something an indie developer could point at their own chatbot, walk away from for two minutes, and come back to with a concrete, evidence-backed answer to "did anything break?"

What it does

DeepAudit AI runs an automated red-team audit against a chatbot target:

• Recon Agent profiles the target's baseline behavior
• Jailbreaker Agent runs a curated library of 15 prompt-injection payloads across seven attack categories, direct override, roleplay, multi-turn escalation, encoding bypass, delimiter confusion, policy violation, and toxicity
• Validator Agent scores every response using GPT-5.6 with structured JSON output: did it leak, what category, how severe (1–5)

The entire run streams live to a dashboard over Server-Sent Events, so you watch each attack fire and resolve in real time. Any finding can be replayed on demand, and a completed audit produces a downloadable PDF vulnerability report with remediation suggestions.

Configurable Target: paste in your own chatbot's system prompt instead of using the built-in demo target, and the same full pipeline, Recon, Jailbreaker, Validator, audits that instead. This proves the tool generalizes beyond one hardcoded scenario: I tested it against a completely different persona (a pet supply store chatbot instead of the built-in retail one) and it correctly profiled, attacked, and scored the new target with no code changes.

Mock mode runs the entire pipeline with realistic simulated responses, so anyone can evaluate the complete product with zero API cost and zero setup beyond npm install.

How I built it

The backend is Node.js, TypeScript, and Express, with an in-memory store (no database, deliberately, to keep the architecture simple and fast to build within a hackathon timeline) that doubles as a pub/sub layer for live event streaming. The frontend is React, TypeScript, Tailwind CSS, and Vite, built around a custom useAuditStream hook that consumes the backend's SSE endpoint.

I built this almost entirely through Codex, working from detailed, upfront specifications for each component, the agent architecture, the payload library, the orchestration pipeline, and later the full frontend, rather than piecemeal prompting. Writing the spec in full before generating code turned out to matter a lot: it forced real architectural decisions (sequential vs. parallel attack execution, why the target had to be a self-hosted sandbox bot rather than an arbitrary third-party endpoint, how to separate "attack" from "judge the attack" into distinct agents) before code existed, rather than discovering them halfway through a build.

Challenges we ran into

A few real ones, worth being honest about:

• A corrupted .env file cost real debugging time, a stray sentence had been typed directly into the middle of the API key value, which produced a misleading "key not configured" error rather than an obviously malformed one.
• A lost Codex session. My very first backend-build session wasn't saved before I realized the hackathon required a /feedback session ID as evidence, a hard lesson in running /feedback at the end of every session from that point forward, not just when we remembered to.
• API billing delays meant most of My development and testing had to happen against simulated responses. I built a dedicated mock mode to keep moving, but my first version of it generated the "leaked" verdict and the fake evidence text independently of each other, producing findings that didn't actually make sense together. Catching and fixing that taught Me to always sanity-check simulated data as closely as real data, not just assume it's "good enough" because it's a placeholder.

What I learned

That a clear, complete specification is worth more than fast, incremental prompting, Codex produced dramatically more coherent, correctly-structured code when given the full architecture upfront versus being redirected piece by piece. I also learned to treat session continuity and feedback capture as part of the build process itself, not an afterthought at submission time.

What's next

Beyond the hackathon, the most valuable next step would be expanding the payload library further and letting users bring their own custom attack payloads alongside the built-in library, turning DeepAudit from a fixed test suite into an extensible one.

Built With

• codex
• express.js
• gpt-5.6
• node.js
• openai-api
• pdfkit
• react
• rest
• server-sent-events
• tailwind-css
• typescript
• vite