Patchlight

In the AI era, code gets written in seconds and breached in minutes. Patchlight is the security layer in between - watching every change and the whole system behind it. Ship faster, without the risk.

Comment

Patchlight - Project Story

Inspiration

Two things happened at the same time, and nobody I knew had a plan for either.

The first is the sheer volume of code. Every developer I know now ships several times more code than they did a year ago, because a model writes most of the first draft. The bottleneck moved. It used to be typing; now it's reading. Human review is still the highest-value check a team has, and it's the first one to quietly degrade under load - it stops being a review and becomes an approval. Someone skims the title, recognises the author, clicks merge.

The second is that the same models that write the code have gotten genuinely good at finding the ways into it. Exploit reasoning is no longer a specialist skill you buy by the day. If an attacker can point a competent model at a public repo and get a working path through a handler, the asymmetry only works out if defenders can point one at their own code first - continuously, cheaply, on every diff.

So the tooling exists. What bothered me was the price shape. Every AI reviewer I looked at was sold per seat: from $24/user/month, $30/dev/month, with line caps of 40–80k per developer per month layered on top. That pricing is copied from human SaaS, where a seat is a person doing work all day. But an AI reviewer doesn't do work per person - it does work per diff. A four-person side project that opens six PRs a month gets billed like four full-time seats. A solo founder who ships 30 PRs one week and none for two months pays every month either way. The people with the least review capacity - small teams, indie builders, anyone whose "second reviewer" is nobody - are exactly the people priced out.

Patchlight is my answer to both: a reviewer on every diff, whoever wrote it, billed for usage instead of seats. 20 free reviews or scans with no card, then a prepaid balance from $10, at roughly a third of a cent per credit. No subscription, no per-developer maths, no line caps.

What I built

Patchlight is a GitHub App with three surfaces around one review core:

  • PR review - the app gets the webhook, fetches the diff, reviews it, and posts inline comments.
  • A security monitor - a scheduled full-codebase scan, not just the diff, because most real vulnerabilities are older than the PR that touches them.
  • A terminal + CI path - patchlight scan --diff reviews the code on your disk right now, uncommitted work included, and a zero-dependency SDK does the same thing in a CI/CD, so you can integrate it in any other pipeline if you are not using GitHub.

Around that: a findings dashboard, weekly reports, an "ask AI" chat over your own findings, and a reverse task check that matches a merged PR back to its Linear issue and reports whether the diff actually did what the ticket asked.

The stack is deliberately cheap to run, because a usage-priced product only works if the floor cost is near zero: a Turborepo/pnpm monorepo, Next.js, a Hono API Worker for webhooks and crons, Drizzle, Better Auth. Workers have no filesystem and no git, so full-repo scans run on a scale-to-zero Google Cloud Run snapshot service.

How AI built it

The architecture spec has a line at the top of the goals section that set the whole tone: the primary developer is an agent swarm. Every decision downstream of that was made to give agents a tight feedback loop - end-to-end TypeScript, schema-as-code, typed env, typecheck → lint → test → build runnable locally in one command, and a rule that infrastructure config is frozen: agents never invent an env var or a binding, they write a line in the README telling me to add it.

And there were a lot of experiments, because I refused to grade my own homework. I forked Code Review Bench - 50 real PRs from Sentry, Grafana, Cal.com, Discourse and Keycloak with human-verified golden comments across Python, Go, TypeScript, Ruby and Java - and wired our reviewer into it as just another contestant. Then I built a second harness for security scanning over a 65-repo corpus, plus a small local web UI.

What I learned

Most good ideas are worth nothing, and you only find out by measuring. The graveyard is longer than the shipping list: chunking the scan input (dead - traded 8 points of precision for 4 of recall), a null/validation checklist (made things actively worse), reverse call-graph context (looked like a huge win until a variance control showed the baseline bounced just as far on a re-run), a semgrep-based SAST pass, micro-agents, higher reasoning effort on the review rail, cheaper base models. Each one felt obviously correct before the numbers came back.

Challenges

The main challenges I faced was not shipping features but rather creating a pipeline, a hardness, a flow that would give AI models and agents the ability to self-check their work run benchmarks and don't cheat on those benchmarks. When you're dealing with a review or a security scanner, you can't really tell whether the thing that was built by the agents is good or not if you can't measure it and the measuring part is the most important part and is probably the only part that shouldn't be built or be able to be changed by agents. As soon as the agent has access to the repo or the code that runs the benchmark, the more and more it is tempted to change the benchmark to increase his chances to make the hardness better or to find ways in which he can restructure the output to adhere to the rules of the benchmark better - this was a big hassle to make sure the auto research that I've built was only keeping code that actually meaningfully improved the score of the benchmark without doing localized fine tuning or downright cheating in order to fit better to the benchmark.

The thing I'd tell anyone building on top of models right now: your product is not the prompt. Your product is the measurement rig that tells you which prompt to keep.

Built With

Share this project:

Updates