PR Impact Analysis

PR Impact Analysis reveals a pull request’s feature-level blast radius and gives developers and QA an evidence-backed checklist of what to verify before merging.

Comment

Inspiration

AI-assisted coding has made it possible to build and ship software faster than ever. But the review process has not caught up.

A pull request can pass CI, look reasonable in a diff, and still affect an important customer flow. Reviewers are often left to trace imports manually and guess which screens, APIs, or workflows deserve testing. Existing code-review feedback is useful for quality and maintainability, but it rarely answers the practical question: what should a developer or QA engineer verify before merging this change?

That is why I built PR Impact Analysis: an evidence-first GitHub App that turns a pull request into a feature-level verification plan.

What it does

PR Impact Analysis analyzes the exact PR diff, traces changed JavaScript and TypeScript code through a deterministic dependency graph, identifies reachable routes and APIs, and posts a single sticky comment on the pull request.

Instead of only saying that a shared component changed, it can say:

  • Test this route.
  • Perform these concrete actions.
  • Expect these outcomes.
  • Here is the dependency path that proves why this route is relevant.

The goal is not to claim that a bug exists. It helps reviewers focus their limited time on the product behavior most worth verifying.

How I built it

The project uses a GitHub App, Express, PostgreSQL-backed durable workers, and a JavaScript/TypeScript dependency graph.

PR diff → deterministic dependency graph → affected routes/APIs
→ prioritized verification targets → GPT-5.6 guidance → sticky PR comment

The graph is the evidence authority. It resolves local dependencies and uses framework adapters for Next.js, React Router, Remix, Express, and tRPC where routes or API bindings can be statically proven.

GPT-5.6 is used after deterministic analysis. It receives only selected changed hunks and graph-proven route context, then produces a concise change summary and source-grounded verification scenarios. Local validation rejects unsupported claims, unknown targets, duplicate scenarios, and generic CI recommendations. GPT-5.6 cannot invent routes, dependencies, or claim that a regression exists.

I used Codex throughout the project to accelerate architecture decisions, graph adapters, webhook and queue reliability, report rendering, tests, debugging, and documentation. I made the key product decisions around the trust boundary: deterministic analysis establishes reachability; AI makes the evidence understandable and actionable.

Challenges I faced

The biggest challenge was avoiding noisy or misleading impact claims. A shared module can have many importers, but showing every importer is not useful to a reviewer. I needed a deterministic policy that prioritizes routes and APIs while keeping analytics, styling, infrastructure, and other technical-only connections visible as evidence rather than promoting them into broad customer-flow checks.

Another challenge was making AI assistance useful without allowing it to overreach. The final design keeps the model bounded to PR-specific source context and validates its output locally. This preserves the usefulness of natural-language guidance without asking reviewers to trust unexplained AI conclusions.

Reliability was also important. Pull requests change quickly, so the system uses durable jobs, idempotency, retries, reconciliation, and one updating sticky comment to keep the visible report aligned with the latest commit.

What I learned

I learned that the most valuable AI developer tools do not need to replace engineering judgment. They can reduce the cost of applying that judgment.

PR Impact Analysis is designed around that idea: use deterministic evidence to establish what is reachable, then use GPT-5.6 to turn that evidence into a clear, human-readable verification plan. The result is a tighter review loop and a more practical answer to the question every reviewer asks before merge:

What should I actually verify?

Built With

Share this project:

Updates