PRoofread

PRoofread turns AI code-review disputes into evidence-backed verdicts and verifiable developer reputation

Comment

Inspiration

I would often push code before lunch and let the CI pipeline run while I was away. When I returned, an AI reviewer had sometimes left findings that lacked context or were simply incorrect. Disputing them meant reopening my laptop, explaining the issue, pushing another change, and waiting through the entire CI cycle again.

I wanted developers to be able to challenge AI review findings from their phones and require the AI to support its claims with evidence rather than treating every generated comment as correct.

What it does

PRoofread is an evidence-based dispute system for AI code reviews.

When a pull request is opened, PRoofread:

  • Reviews the code and creates structured findings.
  • Posts findings on GitHub and sends them through iMessage.
  • Lets the developer challenge a finding directly from their phone.
  • Uses Devin as an independent investigation agent.
  • Allows Devin to ask follow-up questions through the same conversation.
  • Returns a final verdict: DEFEND, WEAKEN, or WITHDRAW.
  • Awards developer reputation based on the outcome.
  • Records reputation events on Solana devnet, making them publicly verifiable.
  • Provides a leaderboard directly through iMessage.

Developers earn reputation points for successfully challenging weak AI findings. Solana makes that reputation history transparent and verifiable.

How we built it

We built PRoofread as a FastAPI backend with SQLite persistence.

The system integrates:

  • GitHub webhooks and APIs for pull-request events, diffs, and comments.
  • OpenAI for the initial code review.
  • Devin for independent investigation of disputed findings.
  • Linq for conversational iMessage notifications and challenges.
  • Solana devnet for publicly verifiable reputation events.

Findings move through a strict state machine:

REVIEWED → CHALLENGED → INVESTIGATING → VERDICT_REACHED

Webhook signatures are verified, duplicate GitHub deliveries are detected, and duplicate reputation awards are prevented. Long-running investigations happen after the webhook is acknowledged so external services do not time out.

Challenges we ran into

Integrating several external systems into one reliable workflow was the biggest challenge.

We encountered:

  • GitHub token permission errors when posting PR comments.
  • Webhook routing and signature-validation issues.
  • Linq sandbox restrictions and conversation-to-finding mapping.
  • Long-running investigation tasks causing webhook timeouts.
  • Devin sessions that paused to request additional context.
  • Devin returning a completed report while its session still appeared to be waiting.
  • Solana SDK compatibility, TLS certificate, wallet-format, faucet, and funding issues.
  • Maintaining idempotency across repeated webhook deliveries.

These problems pushed us to make the workflow observable, stateful, recoverable, and explicit about failures.

Accomplishments that we're proud of

We built a working end-to-end flow across GitHub, OpenAI, Devin, Linq, and Solana.

We are especially proud that:

  • A developer can challenge a code-review finding entirely through iMessage.
  • Devin can ask follow-up questions and receive answers through the same conversation.
  • The AI reviewer is not automatically treated as correct.
  • Final decisions are backed by concrete investigation evidence.
  • Reputation awards cannot be duplicated for the same challenge.
  • Solana reputation transactions contain no source code or private investigation details.
  • The leaderboard is available directly through chat.
  • The integration is covered by mocked automated tests without consuming external API resources.

What we learned

AI code review needs an appeals process.

Models can identify real problems, but they often lack architectural, operational, or business context. Developers should be able to challenge a finding without ignoring it or blindly accepting it.

We also learned that conversational interfaces can make developer tooling more accessible. A developer should not need to return to their laptop simply to explain missing context.

Finally, blockchain was most useful here as a verification layer—not as a token. Solana allows reputation events to be independently verified while the application retains practical queryable state in its database.

What's next for PRoofread

Next, we want to:

  • Replace in-process background tasks with a durable job queue.
  • Move from SQLite to PostgreSQL.
  • Package the integration as a GitHub App.
  • Add authentication and multi-tenant repository isolation.
  • Build a small dashboard for findings, investigations, retries, and reputation.
  • Add GitHub Check annotations with exact file and line references.
  • Improve investigation recovery, monitoring, and error notifications.
  • Build evaluation datasets to measure false-positive and challenge-success rates.
  • Expand the reputation system while keeping private code off-chain.
  • Support additional communication channels after validating the iMessage experience.

Built With

Share this project:

Updates