Litmus

The litmus test for open-source contributions - catches real contributors before they write a line, filters AI slop before it reaches you.

Comment

Inspiration

I kept reading the same story from two completely different kinds of maintainer.

The maintainers of large projects were exhausted. Curl shutting down a six year old bug bounty program because the noise had overwhelmed the signal was the moment it clicked. These are not people who lack tooling. They have CI, linters, review bots and static analysis, and every one of those tools looks at code that has already been written. The waste has already happened by the time any of them speak. The Jazzband Python collective shut down entirely. Forty five percent of maintainers now name burnout as their single biggest challenge.

At the same time I saw the opposite problem up close. Small repositories with genuinely interesting work and zero contributors, and people who wanted to contribute but had no idea where to start or whether they would even be welcome.

These are the same problem seen from two ends. Effort in open source is distributed terribly. It piles onto a handful of famous repositories, often from people who cannot explain what they submitted, while thousands of projects that would love the help get none.

Then I asked what actually separates a real contributor from noise, and it is not the code. It is whether the person can explain their approach when you push back on it. A maintainer works this out in a single comment. That is a conversation, and conversations are exactly what a model with real code reasoning can now have. So I built the thing that has that conversation before the code exists instead of after.

What it does

Litmus is a two sided platform. The test it is named for happens before a single line is written.

For contributors. You connect your GitHub and Litmus reads your real work, the languages and domains where your strongest projects actually live, then matches you to open, unassigned issues you are genuinely equipped to solve. It deliberately steers toward under served repositories instead of the crowded popular ones.

Once you pick up an issue, Litmus reads the relevant slice of that repository, the implicated files, their callers, the tests, the conventions, and gives you a sixty second orientation. Even if you never submit anything you walk away with a free tour of the codebase.

Then you state your intended approach, and this is the heart of the product. Litmus checks that plan against the actual code and surfaces the specific collisions a naive plan would miss. Never generic advice like "consider edge cases", always tied to a real function in this repository. A real example from my demo repo, where a contributor proposed calling delete_document() to evict the oldest entry when storage hits its cap:

Constraint: DocumentStorage uses a non reentrant threading.Lock, and delete_document() acquires that same lock.

Consequence: The first upload that reaches the ceiling will deadlock, and hold the lock indefinitely, blocking retrieval, updates, stats and the cleanup worker.

Evidence: backend/services/document_storage.py:13-19, :21-46, :87-102

How you respond is the test. Revising your plan when shown a constraint is the strongest possible sign you understand what you are doing, and Litmus captures that before any code exists. Justifying your approach with sound reasoning is just as strong. Going silent is a warning.

For maintainers. You connect a repository and Litmus builds a durable structural map of it. You then steer the queue in plain language. Say "I am focused on the auth refactor" or "surface first time friendly issues for newcomers" and Litmus rebuilds the ranking of both open issues and incoming PRs around that intent. Here is the real queue from my demo repository with the focus set to backend storage stability:

Priority Issue Why
94 Cap in memory documents and evict oldest Matches the focus, touches a high dependency module with a locking constraint
68 Upload progress bar stalls at 82% Bounded frontend bug, labeled good first issue, right size for a newcomer
8 🚀 "I can OPTIMIZE your ENTIRE codebase with AI" Cannot point to a single line of this codebase, no acceptance criteria, no scope

The Litmus bot. Flip it on and Litmus works your issue tracker directly. When someone volunteers, it reads the code that issue touches and asks one question that only somebody who opened the file can answer, then reads the reply against the real code and returns a verdict of demonstrated_understanding, partial or non_responsive.

The trust brief. When the PR arrives the maintainer gets a card instead of a cold diff: what the change does in two sentences, the planning trail, a drift check against the scope the plan implied, risk areas, and one sortable verdict. The maintainer still decides everything.

How I built it

The whole thing is deliberately small. A zero dependency Node server and a vanilla JavaScript frontend. No framework, no build step, no node_modules at all. Every minute went into product logic instead of tooling, and the container image builds in seconds.

GPT 5.6 is the engine, not just the assistant. It does contributor matching, orientation, plan collision analysis, the bot's question and its assessment, the trust brief and intent driven ranking. Every call goes through the Responses API with a strict JSON schema, so verdicts are enums and a bad response fails loudly instead of rendering as garbage.

Repo memory is the piece that makes everything else sharp. On first connection Litmus walks the git tree through the GitHub API, selects the most structurally interesting files, resolves import statements across several language families into a real dependency graph, scans for constraint signals like lock discipline, shutdown paths and transaction boundaries, and scores structural risk for every module:

$$R_f = \min\left(100,\ 8 d_f + 12 s_f + 12 \cdot \mathbb{1}[\text{core}] + 12 \cdot \mathbb{1}[\text{convention}] + 4 \cdot \mathbb{1}[\text{test}]\right)$$

where $d_f$ is how many indexed modules import file $f$ and $s_f$ is the count of constraint signals inside it. This is cached against the branch SHA, so the expensive pass runs once and everything after it is fast. It is also what lets Litmus see that two open PRs are heading for a collision in the same fragile module.

The planning trail ties the loop together. Every plan submission is appended to the issue claim with its soundness, the points raised and the evidence cited. When a PR appears Litmus recovers that trail and checks the diff scope against the files the plan implied.

The bot authenticates as a GitHub App by signing a short lived RS256 JWT with Node's built in crypto and exchanging it for a cached installation token, so its comments carry their own identity.

I built Litmus with Codex, which is the honest and slightly recursive part of this submission. I used an AI coding tool to build the tool that helps people contribute well with AI coding tools. It is deployed as a single container on Cloud Run.

Challenges I ran into

Getting the model to shut up. By far the hardest problem was not making it find issues, it was making it stay silent when a plan was genuinely fine. An early version flagged something on every submission, because a model asked to review will always review. A tool that always objects trains people to stop reading it. I fixed this by making zero findings an explicitly valid outcome and by requiring line level evidence for every claim, which made unfounded objections impossible to even express in the schema.

Judging a revision is not the same as judging a plan. When somebody revises after feedback, checking the new plan from scratch throws away the entire point, because the interesting information is whether the revision addressed what was raised. I had to feed the previous plan and previous findings back in and ask specifically whether each earlier point is now resolved, partly resolved or ignored.

A scoring bug that made the whole feature meaningless. My priority queue was returning 1, 2, 3, 4 instead of using the 0 to 100 scale. The ranking order looked correct, so it passed a casual glance, but the scores carried no information about how urgent anything actually was. I only caught it by reading raw API output instead of trusting the UI.

Making a public deployment safe. The moment Litmus goes public it shares one GitHub token with every visitor, which means a stranger could use it to post comments as me on any repository that token can write to. I added a write allowlist so the deployed instance can only comment on the demo repository, and verified it by force enabling the bot on an unrelated repo through the public API and confirming the post was refused before it even spent a model call.

Rate limits shaped the matching design. GitHub's search API is strict, and one search plus a metadata call per candidate hit the limit immediately. Matching now runs a few language targeted searches in parallel, deduplicates, caps candidates per repository so one project cannot flood the shortlist, and degrades to heuristic scoring rather than failing outright.

A three minute demo against forty to a hundred and twenty second model calls. Completely fine for the real product, where a contributor waits a minute and saves a day of wasted work. Brutal for a video.

Accomplishments that I'm proud of

It found a real deadlock that nobody told it about. I wrote a demo issue asking for an eviction policy. Litmus independently worked out that the storage class is guarded by a non reentrant lock, that the obvious implementation would call a method which acquires that same lock, and that the first upload hitting the ceiling would deadlock and freeze the entire service. It cited three separate line ranges. On a later run it also caught a second constraint I had not noticed myself, that the stats route copies only three fields, so a new counter would never surface through the API.

I refused to build an AI detector. This was the most important call I made. It would have been easy to add a "this looks AI generated" score, and it would have been both unreliable and unfair, because plenty of excellent contributors use AI tools well. Litmus judges exactly one thing: can this work explain itself against this codebase. A strong contributor using Codex passes easily. Someone who pasted a diff they do not understand fails at the first question. The spam issue in my demo ranks 8 out of 100 without any claim about who or what wrote it.

The bot tells slop from substance on the same question. Given identical prompts, a buzzword heavy reply scored non_responsive with the note that it never addressed the locking constraint, while a reply that engaged with the lock scored demonstrated_understanding.

Cross PR collision detection works on real repositories. Pointed at a busy open source project, it surfaced six live collisions, including five separate PRs all modifying the same client file.

Zero dependencies. The entire platform runs on node server.js.

What I learned

Specificity has to be a design constraint, not a prompt tweak. My first plan checker produced confident, useless output. The fix was a hard rule enforced by the prompt and the schema together: every finding must depend on a fact visible only in the provided repository context and must cite a path, a symbol and a line range. Requiring the citation is what forced the specificity. Asking politely for it did nothing.

Retrieval beats raw context size. The large context window is genuinely useful, but dumping an entire repository into it produced worse results than selecting sixteen files well. The dependency graph is what made the selection good, because knowing which modules import the file an issue touches means the model sees the callers, and callers are where collisions actually live.

Schemas are how you make a model behave. Strict JSON schemas with enum verdicts removed almost all output variance and made the frontend trivial to write against.

The strongest trust signal is behavioural, not textual. You cannot reliably tell how well someone understands a codebase by reading their prose. You can tell immediately by showing them a constraint and watching whether they adjust.

What's next for Litmus

Persistent storage so repo memory survives across instances rather than living in container memory. A hosted GitHub App anyone can install in one click, so maintainers do not need to run anything themselves. Per repository tuning so maintainers can set how strict the screening question should be, since a first timer friendly project wants a gentler bar than a security library. A feedback loop where maintainers confirm or reject a verdict, so ranking improves against real outcomes instead of my assumptions. And broader language support in the dependency resolver, which today handles the common families well but could go much deeper.

The longer term goal is the part I care most about. Right now Litmus proves a contributor understood the code to the maintainer they are contributing to. That same evidence, built up across many repositories, is a far better picture of what somebody can actually do than a green contribution graph.

Built With

Share this project:

Updates