IntentCheck: AI Pull Request Reviews Against GitHub Issues

AI pull-request reviews that verify code against the GitHub issue it was meant to solve.

Comment

Inspiration

I started IntentCheck with one simple question: a pull request can pass its tests and look perfectly fine, but who checks whether it actually solved the issue it was created for?

The GitHub issue usually explains the real goal, but that context can easily get buried once development and code review begin. I did not want to build another general AI code reviewer. I wanted to build something focused on one specific question:

Did this pull request do what the linked issue asked for?

That became IntentCheck.

What it does

IntentCheck is a Python-based GitHub Action that reviews a pull request against its linked GitHub issue.

When a pull request is opened or updated, IntentCheck:

  1. Finds the issue linked in the pull-request description.
  2. Reads the issue requirements and acceptance criteria.
  3. Examines the files and code changed in the pull request.
  4. Uses GPT-5.6 to compare the implementation with the original requirements.
  5. Posts a review comment showing the verdict, confidence score, supporting evidence, findings, and recommended changes.

Instead of only saying that code looks good or bad, IntentCheck explains which requirements were satisfied and where the evidence appears in the pull request.

I also added a command-line interface so developers can test IntentCheck locally.

How I built it

I built IntentCheck as a reusable Python package and GitHub Action.

The GitHub API is used to retrieve the pull request, linked issue, changed files, and code diff. That information is organized and sent to GPT-5.6 for an evidence-based review. IntentCheck then formats the result and posts it directly on the pull request.

I used Codex throughout the entire build, not just for the initial setup. It helped me plan the project structure, implement the core review pipeline, write tests, debug GitHub Actions failures, prepare the v0.1.0 release, and create a separate repository to test the action properly.

GPT-5.6 powers the actual reasoning inside IntentCheck. It compares the issue requirements with the pull-request evidence and produces the structured review.

Challenges I ran into

The biggest challenge was getting the action to work correctly outside its own repository.

The first version worked locally but failed when I tested it from the separate demo repository because one of the package paths was being resolved relative to the wrong repository. I had to change the installation logic so that it used the GitHub Action’s actual directory.

I also ran into secret configuration and API quota errors during testing. Working through those problems helped me understand how reusable GitHub Actions handle paths, permissions, secrets, and external API calls.

Another challenge was keeping the AI review grounded. I did not want IntentCheck to make vague claims or invent problems. The review prompt had to require evidence from the issue, changed files, code diff, and tests before reporting a finding.

Accomplishments that I’m proud of

I released a working v0.1.0 and tested it end-to-end in a completely separate demo repository.

The released action successfully:

  • Found the linked GitHub issue.
  • Read its requirements.
  • Reviewed the pull-request implementation.
  • Called GPT-5.6.
  • Posted a structured review comment with a 99% confidence pass.

I am also proud that IntentCheck works as both a GitHub Action and a local CLI, while keeping API credentials out of the repository and logs.

What I learned

The biggest thing I learned is that useful AI review depends heavily on good context and clear constraints. Giving a model a code diff is not enough. It also needs to understand why the change was requested and what success is supposed to look like.

I also learned that the model call is only one part of the product. GitHub permissions, workflow configuration, package paths, secret handling, testing, and the final review experience are equally important.

What’s next for IntentCheck

Next, I would like to add GitHub Check annotations, configurable review policies, support for multiple linked issues, stronger test-coverage analysis, and a public GitHub Marketplace release.

The long-term goal is to help teams catch pull requests that are technically clean but still miss the original intent.

Built With

Share this project:

Updates