PRshik

Two PRs pass CI and merge cleanly, then break main together. PRshik flags the risky pair and runs tests on the merged code to prove it.Team PRshik

Comment

Inspiration

Teams now run several developers and coding agents (automated AI tools that write or modify code) on one repo at once. Each branch can be correct on its own and still contradict another branch. Git only checks whether the text merges, and CI tests each branch alone. Nothing checks whether the merged code still works until it is already on the main repository.

What it does

PRshik compares every pair of in-flight branches and proves which pairs break each other. Three agents share one engine:

  1. Sentinel (prediction engine): scores each branch pair from git diffs and a symbol and call-site prefilter. NVIDIA Nemotron, a large language model (LLM), explains the riskiest pairs. PRshik then proves regressions by running each branch's own tests on the merged checkout, plus intent tests that Nemotron generates.

  2. Planner (prevention engine): groups branches into independent zones with no medium- or high-risk pair between them, and orders merges riskiest-first.

  3. Resolver (resolve): reproduces a Git conflict in a throwaway worktree, asks Nemotron for a resolution, and returns a patch with validation results for a human to review.

The dashboard shows each risky pair, both diffs, a pass or fail table for branch A, branch B and A+B, and the failing assertion. An MCP server gives coding agents in code editors and AI-powered coding assistants such as Cursor, Claude Code and similar AI tools the same checks, so an agent can ask "is it safe to merge this branch?" before merging.

Try it: the hosted demo needs no login or keys; click Try demo repo. The bundled repo has two checkout branches. One caps a discount on the pre-tax subtotal, and the other moves tax ahead of the discount. Git merges them cleanly and each branch's tests pass, but the merged checkout returns 200 instead of 198. PRshik flags the pair at 90/100, high and shows the failing test.

How we built it

NVIDIA Nemotron 3 Super on Nebius Token Factory (nvidia/nemotron-3-super-120b-a12b), an LLM accessed via the OpenAI-compatible API. Nebius Token Factory is a cloud platform that provides access to advanced AI models like Nemotron. We chose it for its reliable, scalable infrastructure and seamless integration with our workflow. It assesses risk, writes pytest intent tests, and proposes conflict resolutions. Tavily looks up current documentation for the third-party libraries the changed code uses. Short snippets ground the prompts, and the source links appear in the dashboard and MCP results. Deterministic proof first: a test that one branch added passes on that branch, fails on the merge twice, and does not already fail on the merge-base. This layer needs no LLM and costs nothing. A model risk score on its own never counts as proof. Python and FastAPI run git in subprocesses and run tests in throwaway worktrees with restricted environments and timeouts. A React and TypeScript dashboard sits on top. A stdio MCP server (official Python SDK) calls the engine directly. The public demo is one Render service. Without a Nebius key it runs git-only: git scoring, branch tests and the checkout proof still work.

Challenges we ran into

A convincing explanation from the model does not guarantee that its assessment is correct. Generated tests sometimes failed on their own branch, pinned only the old behaviour, or inspected source code instead of behaviour. In one run, forcing verification on a harmless fixture produced two false confirmations from tests that checked a function's variable names. We added rules so a generated test only confirms when it passes on its own branch, fails on the merge-base, and fails on the merge for a behavioural reason. We also built the deterministic branch-test layer so the core proof does not depend on the model at all.

Cost and latency needed a prefilter, limits on the number of branch pairs analyzed ("pair limits"), restrictions on the length and complexity of model prompts ("bounded prompts"), and effective caching strategies. Tavily results are filtered and trimmed, and never treated as instructions.

Accomplishments that we're proud of:

Four real semantic conflicts from public repos:

  1. daimon: The open source data science agent for Discord and Slack.

  2. KISS: Kernel Interface Standards Suite. The conflict was in its Python tools.

  3. Elastic migration platform: source-agnostic tooling for migrating observability assets.

  4. qwen-code: An open-source AI coding agent that lives in your terminal.

In each case, two pull requests passed CI (continuous integration, the automated tests that run on every pull request) and merged with no textual conflict, yet broke main together.

660-case replay benchmark across 32 Python and TypeScript repos, under a $5 total model spend, with every number published (including misses) in the benchmark results section of the repository. The core proof works with no API keys, so anyone can check it.

What we learned

To measure PRshik at scale, we needed many examples of change pairs that broke each other, and nobody keeps a list of those. So we mined them from project history: if a later fix touched the code of two earlier changes, we counted that pair as broken. This gave us 330 broken and 330 clean pairs across 32 repos.

The shortcut did not hold up. When we checked 49 of the "broken" pairs by hand, only 3 were real interactions; 35 were unrelated and 11 were unclear. A fix that touches two files is usually routine maintenance, not proof that two changes clashed. So the headline scores from the mined labels looked meaningful but mostly measured noise.

We learned two things from this:

  1. Real semantic conflicts are rare and hard to find. Only 3 of 349 hand-labelled pairs were real conflicts. That is why we hunted down and replayed real cases one by one for our Hall of Fame, instead of trusting automatic labels.
  2. Check your ground truth before trusting your metrics. We now publish the mined scores and the hand-checked scores side by side, so readers can see how far apart they are.

What's next for PRshik

  1. Detect semantic or behavioral conflicts that the benchmark missed and improve recall on real code interactions between branches.
  2. Support more programming languages and build systems beyond Python and TypeScript.
  3. Run PRshik as a check on open pull requests from the GitHub review queue, available as a GitHub Action and/or GitHub App integration.
  4. Ensure that every proposed resolution is reviewed by a human and supported by concrete test evidence.

Built With

Share this project:

Updates

Submission history