ProofLine

Inspiration

I kept hitting the same thing with AI-written code. A change looks finished. Functions named right, description clean, tests green. Then you dig in and the requirement was never really met, or a test asserts nothing, or a method just returns a hardcoded value. You catch it in review if someone's paying attention. You catch it after merge if nobody is. I wanted to see the evidence behind a diff before trusting it.

What it does

Proofline reads a public GitHub pull request, commit, or comparison, or a local bundle of files, and shows the evidence behind the change.

• It maps stable requirement IDs like REQ-101 to the exact changed code and test results. When a repo has no formal IDs, it extracts the author's declared change bullets as visibly-labeled CLAIM-001 claims that can never count as strong evidence.
• Each requirement gets one honest state: test evidence found, implementation only, failing test, ambiguous, suggested, or none. Exact-ID matches stay strictly separate from weaker phrase suggestions.
• A separate scan reads the added lines and catches shortcuts an agent leaves behind: TODO placeholders, empty handlers, orphan fixtures, canned mock responses. Each finding names the file, line, impact, and fix.
• An optional model-interpreted pass reads every changed line for the shortcuts those pattern rules can't express—a function that returns a fixed value regardless of its input, an error caught and thrown away, a parameter the body never reads, an assertion that cannot fail. Anything the pattern rules already reported is dropped instead of repeated, so this lane only ever adds signal.
• An optional AI skeptic can challenge weak evidence. You pick specific excerpts, approve the send, and it returns an advisory verdict like "hollow stub" or "vacuous test." It can add a "needs human review" caveat, but it never turns a weak link green.
• Everything exports as Markdown, JSON, or a Mermaid evidence-map diagram.

How I built it

The deterministic core runs entirely in the browser. No account, no database, no analytics, and no model call for the demo, anonymous GitHub analysis, or local import. Anything you load stays in page memory and disappears on refresh.

The one server-side piece is the optional skeptic. It's a small quota-protected Vercel Function that holds a Hugging Face token so the browser never sees a key, scans every excerpt for secrets before it leaves, and caps usage per connection and per day. The model's answer is schema-constrained to a fixed set of verdicts, and if it comes back malformed or cites a line it wasn't given, the result is thrown away rather than shown. The interpreted-integrity pass runs through that same endpoint under its own closed verdict set, so it inherits every one of those guards instead of opening a second door.

I built all of it with Codex as the implementation partner across the whole thing, not just the first prompt: requirements shaping, architecture, the domain logic, the tests, the GitHub integration, accessibility, and a skeptical quality pass at the end. I used GPT-5.6 for part of it, and that's the piece I cite as the 5.6 contribution. (The skeptic's runtime model is a separate hosted model on Hugging Face, not the model I built with.) The calls I kept for myself define what Proofline is: browser-only deterministic analysis, exact-ID evidence separate from phrase guesses, and an AI that's only ever allowed to doubt.

Challenges I ran into

The hard part was adding AI without letting it lie. The entire point of Proofline is that it doesn't overstate what it knows, so a bolted-on "AI says looks good" would have wrecked it. So the skeptic can only ever subtract confidence. Its verdicts are a closed set, its results are advisory, and they never change a deterministic state. Getting that boundary right, and enforcing it in code so a stray model response can't promote evidence, was most of the work.

The interpreted-findings pass taught me that lesson a second time. My first version only read the excerpts already linked to a requirement, so it kept confidently "discovering" the same TODO the regex had flagged a second earlier. It was an AI feature that added nothing but looked busy. Fixing it meant sending every changed line instead of just the linked ones, telling the model plainly what the pattern rules already detect, and dropping any finding whose cited lines were already covered. It only counts as augmentation if it is structurally incapable of repeating what you already knew.

The same restraint runs through the deterministic side: exact IDs make strong links, everything softer is a labeled suggestion, and the tool is willing to say "I don't know."

Accomplishments that I'm proud of

I'm proudest of what Proofline won't do. It never claims the code is correct, it never blends a weak guess into verified evidence, and even the AI can only raise a hand for human review. Before anything reaches the hosted model, you choose the exact excerpts, approve the send, and the tool scans them for secrets and refuses to transmit if it finds one. Shipping an honest review tool, deterministic by default with AI kept on a short leash, felt more valuable than one that looks impressive while quietly overstating what it knows.

What I learned

A review tool is only worth trusting if it's willing to say "I don't know," and that's twice as true once you add a model. Static evidence doesn't prove correctness, and an LLM's opinion proves even less, so the real design work was deciding what each layer is allowed to claim and enforcing it. Most of Proofline is boundaries, not features.

What's next for Proofline

• explicit selection when multiple requirement documents rank equally;
• linked-issue requirement retrieval;
• private repositories through GitHub App/OAuth;
• opt-in saved analyses;
• standard report adapters such as SARIF;
• reusable developer-workflow integrations: extract the analysis engine into a library/package, expose it through a CLI or GitHub/IDE plugin, and offer an installable coding-agent skill so agents and reviewers can run Proofline where changes are created rather than switching to a separate website;
• a selectable advisory model, letting reviewers choose the skeptic from a list of hosted open models;
• cross-file reasoning for the interpreted integrity pass, so a shortcut can be judged against its callers rather than one file at a time.

Built With

• codex
• fast-xml-parser
• github-api
• gpt
• hf
• hugging-face
• huggingfaceinference
• mermaid
• motion
• radix-ui
• react
• react-flow
• react-markdown
• supabase
• typescript
• vercel
• vite
• vitest
• zod