Inspiration

The idea started with a breach that should never have happened. A hardcoded credential sat in a codebase through several rounds of human review, got shipped, and got exploited. Nobody was careless. The problem is structural: a generalist developer reading their own code cannot see what their tooling is silently failing to catch.

That reframed the problem for us. Most small companies do not have a security hire. The market treats that as a coverage problem and sells them more scanners. We think it is a visibility problem. An SME owner running Snyk and Dependabot has no way to answer the only question that matters: what percentage of a real attacker's findings would my current setup miss? Nobody quantifies the gap. So we built the metric that does.

What it does

HoneyPot Wars runs an adversarial security code review on a consented GitHub repository and returns a single headline number: the gap score, the share of attacker-identified findings that the company's existing tooling would not catch.

The pipeline is two agents. An Attacker agent analyses the source and constructs a realistic attack narrative, how a threat actor would chain the vulnerabilities it finds into an actual breach. A Detector agent then reads that narrative and scores each finding against what standard SME tooling (GitHub Secret Scanning, Snyk, SonarQube, Dependabot) would realistically flag. The gap is the value:

gapScore=∣missedFindings∣∣totalFindings∣\text{gapScore} = \frac{|\text{missedFindings}|}{|\text{totalFindings}|}gapScore=∣totalFindings∣∣missedFindings∣ Output is two PDFs generated server-side. A technical report for the developer, with per-finding attacker reasoning, the exploit chain, the detector verdict, and a fix. A plain-English executive report for the business owner: risk framing, business impact, the gap score, no code. One repo submission, two audiences, one number that tells them how exposed they actually are.

How we built it

TypeScript monorepo, three developers, one to two hours each per day. The stack is deliberately boring so the interesting risk stays in the agents: React and Vite on the front end, Express and Socket.IO on the back end, Supabase for auth and storage with row-level security, Zod at every boundary.

The architecture rests on a few invariants we refused to bend:

Attacker runs fully, then Detector scores. No interleaving. The Detector is read-only and never re-reads source, it only sees the Attacker's structured output. This keeps the gap score honest, because the detector is grading findings, not re-discovering them.

Zod is the boundary, not a nicety. Every model response is parsed against a strict schema before anything downstream touches it. A response that fails is retried up to three times, then recorded as a failed extraction. Nothing raw passes through.

Safety is enforced in code, not prompts. Secrets are redacted the moment they are detected and only their metadata is logged. Source never persists; the cloned repo is deleted in a finally block regardless of outcome. No scan begins without a human-set consent flag. CLAUDE.md as persistent context. The whole architecture, the safety rules, the domain model, and the competitive positioning live in one context file the coding agent reads every session. This is what stops AI-generated changes from drifting away from the design across sessions.

Challenges we ran into

The gap score is fragile by construction. The Detector's verdicts are joined back to the Attacker's findings by ID. Early on, Finding IDs were not guaranteed unique, so IDs collided, the join broke, and the gap score came back as a clean 1.0. A perfect number that was completely wrong. Once IDs were made unique the same repo scored 0.8. The lesson stuck: the metric is only as trustworthy as the key it joins on, and a corrupted headline number is worse than no number at all.

Model refusals looked like valid data. On Pass 2, the Attacker would occasionally return a safety-refusal string that happened to satisfy the Zod schema structurally, so it sailed through validation as if it were a real finding. We had to filter refusal patterns explicitly rather than trusting shape validation alone.

The word "penetration test" is a legal landmine. We are building in Singapore, where advertising a licensable security service without a license is an offence under the Cybersecurity Act, regardless of what the tool actually does. We could not call this a pentest anywhere a customer would see it. Every external surface uses "adversarial security code review," and any live code execution stays gated behind written legal clearance. This is not marketing polish, it is a hard constraint that shapes the product.

We kept wanting to build instead of validate. The most persistent challenge was internal. Agentic coding makes it trivial to ship the next feature, which makes it easy to scale execution ahead of evidence. We had to keep forcing the correct sequence: prove the pipeline on a real scan, talk to buyers, then decide features, in that order.

Accomplishments that we're proud of

The gap score framing is ours and, as far as we can tell, nobody else sells it. Competitors sell coverage; we sell the size of the hole in your coverage, which is the thing an SME owner can actually act on.

We got a full end-to-end pipeline working in demo mode without compromising a single safety invariant: no executable payloads in any output, credentials redacted on sight, source deleted after every scan, consent as a hard gate. Building something adversarial by design that is also safe by design was the part we most wanted to get right.

And we made the legal constraint load-bearing rather than an afterthought. The compliance posture is baked into the architecture, not bolted on before a fundraise.

What we learned

Legal framing can be architecture. The pentest-to-review reframe was not a copy change; it drew a hard boundary around what the product is allowed to do and forced us to gate live execution behind real clearance.

Validation before features is a discipline, not a slogan. The tooling that lets you build fast is exactly the tooling that lets you build the wrong thing fast, with total confidence.

Treat schema validation as a crash gate. parse() is where you decide that invalid model output stops, full stop, rather than leaking a plausible-looking error three layers down where it corrupts a headline metric.

What's next for Honeypot

The immediate milestone is the first real live scan: actual model credits against a consented repository, to validate the full pipeline end to end rather than in demo mode. Right behind it is customer discovery, a first outreach wave across two buyer segments (cost-constrained developer-leads and awareness-gap founders) whose interviews will decide feature priorities we have deliberately deferred, starting with how much remediation guidance to bundle.

Further out sits the Dynamic Verifier: sandboxed live exploitation that would let us confirm findings rather than reason about them. That one is gated on written legal clearance under the Cybersecurity Act before a single line gets scaffolded. It is the most powerful thing we could build and the one we are least willing to rush.