Good Faith

Bug bounties that survive the AI-slop era: back your claim with a refundable PayPal deposit, and Claude reproduces it before anyone pays. Real finds get paid in minutes.

Comment

Inspiration

In January 2026 curl, one of the most widely deployed pieces of software on Earth, ended its bug bounty. Its maintainers were spending their time on AI-generated reports that sounded authoritative and described code that did not exist. In April HackerOne paused the Internet Bug Bounty: "The balance between findings and remediation capacity in open source has substantively shifted."

Bounties are how independent researchers get paid for protecting the software everyone runs. Killing them punishes the people doing honest work, including those who use AI well. The problem was never AI; it is claims nobody can check. We wanted to keep the incentive for real findings and remove it for fabricated ones.

What it does

  • Free reports stay free.
  • A bounty claim posts a refundable $20 deposit with PayPal. It is an Orders v2 AUTHORIZE hold, not a charge.
  • Claude triages every report within minutes, treating it as untrusted data. It extracts the claims (version, files, functions, impact, proof of concept), checks that every cited file and function exists at the claimed version, dedupes against verified reports, and replays the HTTP proof of concept against a fresh, network-isolated sandbox seeded with secret canaries. If a canary reaches an account that should not see it, the bug is reproduced: a fact, not an opinion. Instructions aimed at the AI inside a report are detected and count against it.
  • A fixed, conservative policy decides REPRODUCED, DUPLICATE, NEEDS_HUMAN or SLOP. An LLM opinion alone can never make a report slop; it needs hard evidence.
  • A maintainer settles. The AI never moves money. Reproduced: the hold is voided and the bounty is paid with PayPal Payouts, exactly once per report. Duplicate or honest-but-invalid: the hold is voided. Confirmed slop: the deposit is captured into the maintainers' fund, and a dispute is answered with the triage evidence.
  • A public transparency page counts reports, reproductions, slop caught, deposits captured, bounties paid and median time to triage.

Try it: judge mode at https://goodfaith-bounties.vercel.app/demo settles three real claims end to end on the PayPal sandbox in about two minutes. Video: https://youtu.be/NekFe7xzLMs

How we built it

A TypeScript monorepo:

  • packages/paypal: a typed PayPal REST client with a faithful in-memory mock, tested live against the sandbox: Orders v2 (AUTHORIZE and CAPTURE), Payments v2 (void and capture of authorizations), Payouts v1, Disputes (provide-evidence), Webhooks (signature verification), Vault, Transaction Search. Every money call carries a fixed PayPal-Request-Id, so a retry is a replay.
  • packages/llm: provider-agnostic structured output over the Anthropic API (Claude Opus 5.5), with Gemini as a fallback and untrusted-content spotlighting.
  • packages/triage: the pipeline (claims → version → code references → duplicates → sandbox replay → assessment → verdict), the deterministic code-reference checks and the verdict policy, plus the evals.
  • packages/target: Lantern, a small self-hosted notes app written for this project in two real versions. 1.4.0 has an IDOR (CWE-639), a path traversal (CWE-22) and a mass assignment of role (CWE-915); 1.4.1 fixes them. Every replay builds a fresh in-memory instance with random canaries and no network.
  • apps/web: Next.js 16. The researcher's report form with the PayPal JS SDK v6 button, a live triage timeline streamed to the status page, the maintainer console (AG Grid) with pay / release / capture behind confirmations that list the exact PayPal calls, the transparency page (AG Charts), sponsor funding, webhooks, and judge mode. Hosted on Vercel with Upstash Redis; settlement is a compare-and-set so racing instances resolve a deposit exactly once.

Everything runs with zero keys (mock PayPal and a scripted model, labelled MOCK in the UI). The hosted demo uses the real PayPal sandbox and real Claude.

Challenges we ran into

  • Making "reproduced" a fact. An LLM saying a bug is real is an opinion. We seed every sandbox run with canaries (Bob's private note, secrets.env, an admin-only list) and replay the proof of concept against a fresh instance. Either a canary reaches an account that should not see it, or it doesn't.
  • Never charging an honest researcher by machine. The policy is deliberately asymmetric: a canary hit is proof, but SLOP needs hard evidence (a cited function or version that does not exist, or an instruction aimed at the AI) and a high slop score, and a maintainer still confirms the capture. On curl's real reports, 8 of 10 published slop reports go to a human rather than being punished automatically, because they cite real code. We chose that over a better-looking number.
  • Reading reports as untrusted input. Slop farms will write for the triager. Report text is spotlighted as data in every prompt, hidden instructions count against the report, and a report with a real PoC still reproduces even when it also contains an injection.
  • Serverless time limits. Triage runs inside a 300-second function. Every model call takes an abort signal, timeouts are budgeted per call, and if the model stalls the run finishes on the deterministic checks in a labelled degraded mode that can never say SLOP.
  • Exactly-once money. Payouts use the report id as sender_batch_id, every PayPal call carries a fixed request id, and settlement is a compare-and-set on the store, so a retry or a second click can never pay or capture twice.

Accomplishments that we're proud of

  • The complete loop runs live on the PayPal sandbox: hold → triage → void plus Payout, void, or capture, with real PayPal ids in the console.
  • 30 of 30 reports triaged correctly by Claude Opus 5.5 on our Lantern corpus (8 real bugs, 11 slop, 8 for human review, 3 duplicates), with zero honest reports called slop; the same 30 of 30 with the zero-key scripted model.
  • On curl's real, published reports: 0 of 5 real CVEs flagged as slop. 2 of 10 published AI-slop reports caught on hard evidence; the other 8 cite real code and go to a human.
  • Lantern: a real two-version app with three CWE-mapped bugs, so the demo reproduces real vulnerabilities rather than mocked ones.
  • A maintainer's median time from deposit to verdict on the hosted app: seconds, not days.

What we learned

  • Authorization holds map cleanly onto deposits: AUTHORIZE, then void or capture, with the 3-day honor period becoming the product's triage promise. The approval sheet still says "Complete Purchase", which is the one place buyers may hesitate.
  • Idempotency is PayPal's best under-advertised feature: PayPal-Request-Id and sender_batch_id turned "pay exactly once" from a hard distributed-systems problem into a convention.
  • Most AI slop about real projects cites real code. Fabrication checks catch the lazy fraction; the rest needs reproduction or a human, so a product that promises to "catch all slop" is itself slop.

What's next for Good Faith

  • A GitHub App that opens the program from a repository's security policy and files advisories for reproduced reports.
  • More target types: containers for non-HTTP projects, so the sandbox replay covers CLIs and libraries.
  • Researcher reputation across programs, with trusted researchers skipping the deposit.
  • Deposits funded from a saved PayPal wallet for researchers who opt in, and bounty pools funded by sponsors through PayPal Checkout.

Built With

  • ag-charts
  • ag-grid
  • anthropic-api
  • claude
  • hono
  • next.js
  • paypal
  • paypal-disputes
  • paypal-js-sdk-v6
  • paypal-orders-v2
  • paypal-payouts
  • paypal-webhooks
  • react
  • tailwindcss
  • typescript
  • upstash-redis
  • vercel
  • vitest
  • zod
Share this project:

Updates

Submission history