MergeSignal — Trust for Every Pull Request

GPT‑5.6 turns GitHub history, code quality, and repo-specific expertise into an evidence-backed contributor signal—helping maintainers filter low-quality AI PR spam.

Comment

MergeSignal — Stop low-quality PRs early!

An evidence-backed contributor reputation signal, built from public GitHub history.

Inspiration

The open-source contribution model is being overwhelmed:

  • March 18, 2026: Hugging Face CEO Clement Delangue reported approximately one new pull request every three minutes across its largest repositories, making GitHub nearly unusable. Source
  • May 27, 2026: Mark Ajzenstadt described AI-generated code as polished and idiomatic while still being architecturally wrong—forcing scarce senior engineers to verify it. Source
  • July 15–17, 2026: Storybook and Jonathan H. Wage described reviewing, testing, and merging as the new development bottleneck.
  • Concrete fallout: curl shut down its bug bounty after junk submissions surged, while Ghostty introduced strict rules allowing maintainers to immediately close unverified AI-assisted contributions.

When a repository receives a new PR every few minutes, running another LLM over every diff is not enough. Maintainers first need to know:

  • Is this account three days old or ten years old?
  • Has this person contributed consistently?
  • Have independent maintainers merged their work before?
  • Do they respond constructively to review feedback?
  • Is their experience relevant to this repository?
  • Does the history look authentic or artificially manufactured?

Existing code-review apps analyze the patch. MergeSignal focuses on the missing layer: the reputation and demonstrated track record of the contributor behind it.

What it does

When someone opens a pull request, MergeSignal builds a repository-specific Contributor Reputation Report from their public GitHub history.

It evaluates:

  • Account tenure: age and continuity of the account
  • Open-source history: substantive contributions to independent projects
  • Merge record: successfully merged pull requests and acceptance patterns
  • Consistency: sustained work over time rather than sudden activity bursts
  • Collaboration: reviews, issue participation, and responses to maintainer feedback
  • Relevant experience: prior work in related repositories, languages, and domains
  • Evidence confidence: how much public information supports the result
  • Gaming indicators: self-merged work, repetitive low-value changes, or suspicious contribution patterns

The output explains every signal and links to its supporting GitHub evidence.

For example:

  • A seven-year-old account with regular contributions and merged PRs across independent projects receives a strong, high-confidence reputation signal.
  • A newly created account that opened dozens of similar PRs in one day receives a limited-evidence warning.
  • A genuine newcomer with one focused contribution is marked “insufficient history”, not untrustworthy.
  • An established contributor may have strong general reputation but limited experience relevant to the current repository.

MergeSignal can still run lightweight checks on the current patch—CI status, size, scope, and repository-policy compliance—but code analysis supports the reputation signal rather than defining the product.

How we built it

MergeSignal runs as a GitHub App:

  1. A webhook receives a pull-request event.
  2. GitHub REST and GraphQL APIs collect the contributor’s public account, contribution, pull-request, review, and repository history.
  3. A deterministic scoring engine separates independently validated work from self-owned or self-merged activity.
  4. GPT-5.6 evaluates how relevant that history is to the current repository and produces a concise explanation.
  5. MergeSignal publishes the result through GitHub Checks.

The underlying history remains deterministic and inspectable. GPT-5.6 contextualizes the evidence; it does not invent it.

We used Codex for architecture, implementation, test generation, scoring analysis, and adversarial testing. Our stack includes TypeScript, Next.js, Octokit, GitHub Apps, GitHub Checks, the OpenAI Responses API, GPT-5.6, PostgreSQL, Drizzle, Zod, Vitest, Playwright, and Vercel.

Challenges we ran into

  • Fairness to newcomers: no history must mean “unknown,” not “bad.”
  • Gaming resistance: old accounts, contribution counts, and self-merged PRs can manufacture superficial credibility.
  • Relevance: ten merged documentation PRs do not automatically demonstrate security expertise.
  • Incomplete data: private work and off-platform experience are invisible.
  • GitHub limits: contribution history must be collected efficiently within API rate limits.
  • Explainability: maintainers must be able to inspect why every signal was assigned.

We addressed these by separating signal dimensions, discounting self-controlled activity, exposing confidence, and linking every conclusion to public evidence.

Accomplishments that we're proud of

  • Making contributor reputation evidence-backed instead of popularity-based
  • Separating independently merged work from self-reported activity
  • Measuring consistency and collaboration—not just contribution volume
  • Creating repository-specific reputation instead of one permanent global score
  • Giving newcomers a fair “limited evidence” state
  • Keeping every conclusion transparent and inspectable
  • Delivering the result inside the existing GitHub pull-request workflow

MergeSignal does not decide whether someone is trustworthy. It shows maintainers the public evidence needed to make that decision faster.

What we learned

Reputation is not account age, follower count, or a green contribution graph.

The strongest signals are repeated examples of useful work accepted by independent maintainers, sustained participation, constructive review behavior, and experience relevant to the current repository.

We also learned:

  • Account age is context, not proof.
  • Consistency matters more than raw volume.
  • Merged external PRs are stronger evidence than self-merged commits.
  • Missing history must never become a negative judgment.
  • GPT-5.6 is best used to explain relevance—not as the source of truth.
  • Reputation should prioritize review, never replace review.

What's next for MergeSignal — Trust for Every Pull Request

Next, we will calibrate the reputation model with real maintainers and contribution histories.

We plan to add:

  • Repository-specific signal weights
  • Maintainer feedback and calibration
  • Stronger anti-gaming and coordinated-account detection
  • Organization-level contributor intelligence
  • Contributor-facing reports with evidence correction
  • Reputation changes over time
  • A preflight view for contributors before they open a PR

As agents make submitting code nearly free, MergeSignal helps maintainers identify the contributors whose demonstrated history makes their work worth reviewing first.

Built With

Share this project:

Updates