AI Code Reviewer

This is a reviewer of code generated by AI

Comment

Inspiration

AI makes writing code faster, but understanding and verifying that code still takes significant effort. A small change can affect authentication, payments, validation, or database behavior in ways that are easy to overlook.

AI Code Reviewer grew from a question: could an AI review tool explain what changed, identify where human attention matters most, and provide evidence that reviewers can inspect?

What it does

AI Code Reviewer investigates public GitHub pull requests and turns their changes into a prioritized review report.

It combines deterministic risk scoring with AI-assisted behavioral analysis to identify potential issues, explain their impact, and propose targeted verification tests. Test generation uses source files, nearby tests, configuration, and dependencies retrieved from the recorded commit.

When an execution provider is configured, the application can run repository checks and generated tests in an isolated sandbox. Reports distinguish executed results, static observations, unverified risks, and unavailable checks.

Reviewers can explore diffs, inspect evidence, record decisions and notes, compare analyses, and export findings. Investigations support saved checkpoints, pause/resume, and recovery after refreshes or server restarts.

A credential-free demo executes real assertions against a bundled fixture containing a deliberate password-reset defect.

How we built it

The application uses Next.js, React, and TypeScript for the dashboard and server APIs. Octokit retrieves GitHub pull request metadata and diffs, while a deterministic engine categorizes files and calculates explainable risk scores.

Nebius Token Factory provides structured model inference with an explicitly selected NVIDIA model. Zod validates model responses and persisted reports.

Execution adapters support Docker and Nebius ConTree sandboxes. Local filesystem storage preserves reports, reviewer decisions, and investigation checkpoints.

Vitest tests cover analysis, scoring, validation, security boundaries, streaming, persistence, and recovery behavior.

Challenges we ran into

A major challenge was preventing uncertain AI output from appearing as verified evidence. A generated test can fail because its imports, fixtures, or assumptions are wrong, so a failure alone cannot establish a product defect.

Generating useful tests also required more context than a diff provides. We added commit-pinned source retrieval, relevant test examples, dependency context, and explicit handling of missing information.

Interrupted execution created another challenge. Checkpoints now preserve confirmed work, while resumed verification prepares a fresh sandbox and checks that saved evidence matches the recorded commits and test source.

Accomplishments that we're proud of

  • Built a complete workflow from pull request input to an inspectable review report.
  • Added explainable risk scoring and evidence-linked findings.
  • Improved generated tests with repository context and recorded provenance.
  • Implemented draft protection, conflict handling, and resumable investigations.
  • Reached 509 passing automated tests, with two optional sandbox integrations skipped.
  • Verified browser refresh recovery, pause/resume, and recovery after a server restart.

What we learned

Reliable AI tools need clear boundaries around what they know, what they inferred, and what they actually executed.

Better repository context improves the foundation for test generation, but human inspection remains essential. Passing application tests also does not replace evaluating the complete workflow on real repositories.

We learned that recovery, persistence, and honest reporting of limitations are central parts of a useful developer tool.

What's next for AI Code Reviewer

The next priority is evaluating the complete live workflow on representative real pull requests, measuring finding usefulness, false positives, and generated-test execution success.

Further improvements include a dashboard for interrupted runs, checkpoint retention controls, broader framework and monorepo support, and stronger deployment monitoring.

The current application targets a single-owner workspace. Future work could extend it with distributed execution workers and GitHub workflow integration.

Built With

Share this project:

Updates

Submission history