Codex Audit

Inspiration

AI coding agents like Codex are now writing meaningful chunks of production code, but the way teams review that code hasn't caught up. A pull request from an AI agent still gets reviewed the same way a human's PR does: someone scrolls through a diff and hopes they catch anything risky. That doesn't scale, and it defeats the point of using an agent in the first place if a human still has to manually re-read every line to feel safe merging it.

We wanted to build something that closes that gap, a tool that reviews the reviewer's blind spot: what actually changed, why, and how risky it is, distilled into something readable in seconds instead of minutes.

What it does

Codex Audit takes a code diff or an AI coding-agent session transcript and turns it into a structured, plain-English risk report:

• Risk score (Low, Medium, or High) based on real signals such as hardcoded secrets, new dependencies, changes to auth or security sensitive files, deleted tests, and oversized deletions
• Decision log, a reconstructed, plain-English explanation of why each major change was likely made
• Reviewer checklist, specific and prioritized items a human should double check before approving
• One-click Markdown export, so the report can be dropped straight into a pull request description or a compliance record

The goal isn't to replace human review, it's to make that review fast, targeted, and genuinely trustworthy.

How we built it

We built Codex Audit as a full-stack Next.js application, styled with Tailwind for a clean, dark-mode dashboard experience. The core logic lives in a single API route that sends the pasted diff to GPT-5.6 with a strict system prompt instructing it to reason like a senior code reviewer and return structured JSON, so the frontend can render it as a real dashboard instead of a wall of text.

We used Codex to build the project end to end from a single detailed prompt covering project scaffolding, the API route, the JSON schema enforcement, the results dashboard UI, error handling for edge cases, and the Markdown export feature. From there we ran follow up prompts specifically to stress test the app against bad input, polish the UI so it felt like a finished product rather than a prototype, and rewrite the copy so it reads naturally rather than like generic placeholder text.

One concrete example: when we stress tested the app with empty input and malformed text, Codex identified that the API route would fail silently in certain cases and rewrote the error handling so the app always shows a clear message instead of a blank screen or a raw server error.

Challenges we ran into

• Getting the model to reliably return strict, parseable JSON every time, even on messy or unusual input, took several rounds of prompt refinement
• Local environment setup ate into build time. We hit a PowerShell script execution policy block that stopped npm from running at all, and had to correctly locate the actual project directory before commands would work
• Handling edge cases gracefully, including empty input, non-code text, and very large diffs, without the app crashing or exposing a raw error
• Making sure API keys and environment variables were never committed to the public repository, which meant setting up a proper .gitignore and an .env.example file before pushing anything

What we learned

Reviewing AI-generated code isn't just about spotting bugs, it's about understanding intent. A lot of what makes a change risky isn't visible in the diff alone, it's the context of why it was made. Getting the model to reconstruct that context in plain English turned out to matter more than the raw risk score itself. We also learned how much of the real work in shipping a usable AI tool is in the surrounding details: error handling, environment setup, and making sure the demo path actually works for someone who has never seen the project before.

What's next for Codex Audit

• GitHub integration to automatically run an audit as a pull request check
• Confidence scoring per flag, so reviewers know how certain the analysis is
• Support for analyzing full commit history, not just a single diff
• Team dashboards to track risk trends across a codebase over time

Built With

• codex
• gpt-5
• javascript
• nextjs
• node.js
• openai
• react
• tailwindcss
• typescript