Agent Diff Auditor

Most AI reviewers check if the code is clean. This one checks if it's doing what you actually asked.

Comment

Inspiration

Most AI code review tools check whether code is well-written. None of them check whether it's doing what it was actually asked to do. An AI agent can write clean, well-tested, perfectly styled code that silently does something completely different from the task it was given — and every quality-focused reviewer will wave it through. That gap, between "the code looks fine" and "the code does what we think it does," is what Agent Diff Auditor closes.

What it does

Agent Diff Auditor takes a unified git diff and an optional description of the task the agent was given, then runs parallel analyses powered by GPT-5.6:

  • Consistency check between the stated task and the real diff — flagging when a change is out of scope for what was asked. This is the core of the tool: everything else exists to support this one judgment.
  • Risk scoring per file (low / medium / high / unknown), grounded strictly in the diff content — never a guess based on a filename alone
  • Plain-language intent summary of what actually changed, with exact line citations
  • Test generation for high-risk files, producing real runnable test code (Vitest) rather than a placeholder
  • "Explain this" — select any line in the diff for a focused, scoped explanation

General-purpose code review tools score quality: style, bugs, test coverage, readability. Agent Diff Auditor scores something upstream of that — whether the diff actually matches the task it claims to solve. A change can be clean, idiomatic, and pass every linter, and still silently do the wrong thing. That's the failure mode most review tools aren't built to catch, and it's the one this tool is built around.

Try it live: a diff that silently removes an authorization check while claiming to "improve performance" gets flagged HIGH risk, an explicit task-mismatch warning, and a generated test that proves the vulnerability — all in a few seconds. This is the exact scenario shown in the demo video.

Why not just paste the diff into a chat window? Because a free-text answer from an LLM can hide a hallucination behind confident-sounding prose. Every claim Agent Diff Auditor makes is schema-constrained, tied to a cited line range in the diff, and re-validated after generation — there's no open-ended response for a wrong answer to hide in. If the model can't ground a judgment in the actual diff, it's required to say "unknown" instead of guessing.

How I built it

I built the entire project with Codex, running on GPT-5.6, over one week. The architecture follows the same pattern across all five analysis features: a Zod schema for validation, a service layer using OpenAI's Responses API with structured outputs (text.format + JSON Schema), and a Next.js API route. Every model response is revalidated with Zod even though structured outputs are already schema-constrained — defense in depth against a model returning something technically valid but semantically wrong.

Codex scaffolded this pattern quickly and consistently across features, wrote the diff parser with unit tests for edge cases (new files, pure deletions), and helped trace down real bugs — including a script that exited silently because of a CommonJS/top-level-await incompatibility with tsx.

To develop and test against real LLM calls without burning API budget, I built a swappable provider architecture: a cost-efficient OpenAI-compatible provider for local development, and OpenAI/GPT-5.6 for production, switched with a single environment variable. The public demo deployed here runs on the development provider for cost reasons — the code path for GPT-5.6 is identical and one env var away. This wasn't just a budget workaround; it's also what made it possible to test the full pipeline dozens of times during development without worrying about cost, which directly shaped how much I could iterate on prompt design.

Challenges I ran into

  • Anti-hallucination prompt design: every system prompt explicitly forbids referencing anything not literally present in the diff, and requires an "unknown" fallback instead of a guess. This mattered more than any other single design decision for making the tool trustworthy rather than just plausible-sounding — a risk-analysis tool that hallucinates is worse than no tool at all.
  • Fixture corruption: hand-copying git diffs into test fixtures repeatedly lost +/- prefixes and leading whitespace, silently breaking test validity. Fixed by generating fixtures directly from git diff instead of typing them by hand — a small process change that eliminated an entire class of bugs.
  • A genuinely confusing intermittent bug: parallel API calls would sometimes resolve, sometimes hang forever with no error. Root-caused (with Codex) to a specific free-tier model being unreliable under concurrency — switching models fixed it outright, and it taught me to isolate variables with curl before assuming the bug was in my own code.
  • Working within a real budget: minimum API top-ups made local development costly relative to actual per-call cost, so the swappable-provider design above became a real architectural decision, not just a nice-to-have.

What I learned

That the review problem for agent-written code isn't really "is the code correct" — it's "does the diff still do what I think it does." Most of the value came from cross-checking stated intent against actual changes, not from the risk score alone. A risk score tells you what might be dangerous; the consistency check tells you why it's there in the first place — and that second question is the one a tired human reviewer is most likely to skip. It's also the question most existing review tooling doesn't ask at all, because it requires knowing the task, not just the code.

What's next

  • Multi-file cross-referencing, to catch risk that only appears when two files change together (e.g. a permission check removed in one file while a new caller is added in another)
  • A running history of past audits per repository, to spot patterns over time — repeated "performance" tasks that keep touching auth code, for example
  • Turning the example GitHub Actions workflow included in this repo into a proper GitHub App, so teams can install it without maintaining their own CI configuration

Built With

Share this project:

Updates