Inspiration

AI coding tools let anyone ship a working web app in an afternoon, but the people shipping them often can't read the code the AI wrote. A vibe-coded signup page might store a user's name and date of birth in plaintext, or commit a live API key to a public repo, and the builder would never know. Existing security scanners are built for developers who can read a CVE report. We wanted a checkpoint for everyone else: a tool that tells you, in plain English, what's wrong with your AI-generated code before you publish it, and that runs where they already work.

What it does

Chaos Monkey scans AI-generated projects and produces explainable, severity-ranked reports. Each finding includes the file and line, the evidence from your actual code, the matching CWE, a plain-English explanation of why it matters, and a suggested fix.

Detects SQL injection, XSS, command injection, unsafe deserialization, weak cryptography, insecure configuration, exposed secrets and API keys, unsafe browser storage, and PII exposure
Runs locally: the app is configured to refuse any non-localhost model endpoint, so your code isn't sent to a cloud AI service
Works with AI off: detection is done by deterministic rules, and the local LLM only adds explanations and fix suggestions
Sandbox verification: findings are tested against the running app using planted fake data, so a "confirmed" badge means we actually extracted a canary value, not just guessed
Claude Code plugin: run one command inside Claude Code to scan the current project, and optionally block risky pushes automatically

How we built it

Backend: FastAPI, with a local SQLite store
Frontend: a local web app with a folder-upload scanner
Detection: a rule-based scanner (expanded from 29 rules to [N]) that runs in both the backend and the frontend upload flow
Knowledge base: CWE entries embedded and retrieved so each finding is grounded in the real weakness description and mitigation
Local AI: Ollama, with temperature 0, a fixed seed, top-k 1, and schema-validated JSON output so results are reproducible. Responses are cached against the model digest and input
Safety design: scanned code is treated as untrusted data in prompts, and the LLM can add explanations but can't add or remove findings
Sandbox: the target app runs in an isolated Docker container with no network access, and fixed, versioned probes check each finding
Claude Code plugin: packaged with a .claude-plugin/plugin.json manifest plus [skills / hooks / MCP server: list only what exists]. /chaosmonkey:enable calls the local backend and returns the report in the session. A pre-push hook scans before git push and flags exposed secrets or critical findings
We used Claude Code to help generate and iterate on parts of the codebase, and we reviewed and tested the results ourselves.

Why the plugin matters for usability

The people most likely to ship vulnerable code are the ones who never open a separate security tool, so the check has to live where they already are.

Without the plugin: N steps, e.g. start the backend, open the web app, upload the folder, read the report
With the plugin: [N steps, e.g. type one command]
If built Fix loop: Claude Code can read a finding, apply the suggested fix, and re-run the scan to confirm it's gone
What happens if the backend or Ollama isn't running, e.g. a clear message instead of a crash

Challenges we ran into

False positives. A scanner that cries wolf gets ignored. Flagging every use of innerHTML or the word SELECT was useless, so we tuned rules to look for real patterns (untrusted input reaching a dangerous sink) and built a labeled set of vulnerable and safe samples to measure precision and recall: [results]
Making an LLM deterministic. Temperature 0 isn't enough on its own. We pinned the model, seed, and options, and tested the same input repeatedly: [your actual result]. We learned to say "reproducible on the same setup" instead of overclaiming
Honest privacy claims. Our check validates that the configured model endpoint is local. We tightened it to be enforced per request and describe exactly what it does and doesn't guarantee
Bad fixes. Our own suggested fixes included broken code, including a faulty bcrypt example. A security tool that gives wrong fixes is worse than none, so we added tests that syntax-check every suggested fix
Keeping rules consistent between the backend and the frontend upload scanner
Scoping the sandbox. Fully autonomous exploit generation is flaky and risky, so we limited it to verifying specific findings with fixed probes on a narrow set of app types
Plugin integration: getting the plugin to find the local backend

Accomplishments that we're proud of

Getting Claude plugin to work effectively
13 scans produced identical output
A plain-English report a non-programmer can act on
Going from generated code to a security report with a single command
Prioritizing privacy by making LLM locally run

What we learned

Detection should be deterministic, and the LLM should only explain. That split made the product more trustworthy and testable
"Privacy" and "deterministic" are easy to claim and hard to prove, so we wrote tests instead of slogans
Evaluation sets with safe code matter as much as vulnerable code
A security tool only helps if people actually run it, so where it lives matters as much as what it detects
Scoping a hard problem down made it buildable in a hackathon