Complaint Replay

Can't reproduce? Now you can. Replays a user's bug report in a real browser under their own conditions, proves it, fixes it with NVIDIA Nemotron, and guards it with a test.

Comment

Inspiration

A customer writes "OTP never comes!", or "الزر لا يعمل", or "I use a screen reader and can't find the cart". The developer tries it on a fast laptop with good Wi-Fi. It works, so the complaint is closed as "can't reproduce" and the customer leaves.

That is not rare. Studies of real bug trackers find 13-24% of bug reports are never reproduced, and a large share fail because the user's device, network or settings differ from the developer's. Small teams feel this most: no QA team, no device lab, and a support inbox full of complaints in languages nobody on the team reads.

What it does

Complaint Replay turns a user's complaint into a pull request: proof of the bug, the fix, and a test that stops it from coming back.

  1. Hears. A one-line widget adds "Report a problem" to any site, in 17 languages, typed or spoken. With each complaint it records the user's real conditions: screen, device, network, language, timezone, dark mode, reduced motion. It also catches silent frustration (rage clicks, page errors, failed requests), and takes complaints from GitHub issues, support-desk email and Sentry.
  2. Reproduces. In the team's own GitHub Actions, an AI twin opens a real browser under those exact conditions and repeats what the user did. If the user says they use a screen reader or only a keyboard, in any language, the twin does too. iPhone users replay in Safari's own engine, older users at their own slower pace, and commuters with a signal that drops mid-payment.
  3. Proves. Evidence comes only from code watching the browser: failed requests, a button pushed off a phone screen, an English error shown to a Hindi user, a control a screen reader announces only as "button", the same payment sent twice. The model may cite evidence; it cannot invent it. For screen-reader bugs, the developer can press Listen and hear the page the way a blind user does.
  4. Fixes. NVIDIA Nemotron reads the evidence, the code that ran when the bug showed, and the relevant source files, and writes the smallest fix on a new branch.
  5. Guards. The steps that triggered the bug run again on the fix, by code, with no model. They become a test that must pass on the fix and fail on the old code, and it runs in CI on every later pull request.
  6. Closes the loop. A pull request arrives with the evidence, the guard, and two short films of the user's steps: the bug, then the same steps on the fix. The person who complained sees "fixed" in the widget, with a link to watch it working.

Coding agents can drive it too: crp mcp gives Claude Code, Cursor or Copilot tools to replay a complaint and run the guards, so an agent can work on a proven bug until its guard passes.

The source code never leaves the team's CI. Teams set it up in about ten minutes: one command writes the workflow, then one repository secret and, for the widget, one script tag.

How I built it

  • NVIDIA Nemotron 3 Super, on NVIDIA's free API catalog (the same client calls Nebius Token Factory when given a Nebius key), in four roles with different settings. The navigator picks one click or keystroke at a time with thinking off, because "tap Pay" needs speed. A judge, plus a second skeptic question per item, decides whether each piece of evidence shows the problem the user described. A reader picks out what users say about themselves ("I'm blind", "मुझे बड़े अक्षर चाहिए"). The fixer reads evidence and code with reasoning on.
  • Nemotron Embed groups the same bug reported in different languages, so one bug gets one investigation and a count of how many people it hurts.
  • Playwright drives real Chromium, and WebKit (Safari's engine) for iPhone users. Conditions are enforced, not role-played: 207 device profiles, eight network profiles from 4G to "drops after load", locale and timezone, right-to-left, forced colors, 200% text, an accessibility-tree-only view for the screen-reader twin, and only Tab, Enter, Space, Escape and arrows for the keyboard twin.
  • axe-core and fastText language ID give hard accessibility and language evidence on CPU.
  • GitHub Actions runs every replay in the customer's own CI. A FastAPI server handles projects, widget keys, PII masking (including Devanagari digits, PAN and Aadhaar), signed case tokens and retention. (Sign-in through a GitHub App made from a manifest is built and unit-tested but not yet tried against a live App.) React, Vite and Tailwind for the dashboard: setup wizard, an impact-ranked board, case detail with evidence, diff, proof and guard.
  • An MCP server (crp mcp) exposes replays and guards to coding agents; the films are captioned GIFs drawn with Pillow; the Listen button uses the browser's own speech engine, so it costs nothing.
  • Self-host with Docker Compose, host on Render's free plan with nothing to fill in but keys, or bring your own model through any OpenAI-compatible server.

Results, misses included

  • On someone else's code: a public React shopping cart I did not write, copied unchanged. Three real accessibility bugs, three complaints, three guarded pull requests, one of them through the full loop from widget to pull request to "fixed" in the widget. A fourth complaint, written in Hindi, was also fixed and guarded.
  • On real bugs from a real product: 8 bugs that the developers of Actual Budget (an open-source budgeting app) fixed in the last two months. I put each bug back by reversing the real fix, gave Complaint Replay only a user-style complaint, and counted it solved only if it fixed the bug in the same file the real developers changed. Result in the latest run: 1 of 8, a date picker that never opened on Android, fixed in the same shared component the developers changed (my fix differs from theirs; the table says how). Results vary between runs, and 8 bugs is a small sample. These 8 bugs were also my development set: I changed the engine after each of 15 runs, so read 1 of 8 as "it can", not as a rate. Full table, every miss included, and how tuned it is: benchmarks/real-bugs.md in the repo.
  • On planted bugs: 7 bugs, 3 runs each: 21/21 reproduced, 20/21 fixed and guarded, 0 false alarms on the clean app.
  • Languages: on 400 complaints in 20 languages written by Nemotron (synthetic, said plainly), the same bug in another language is matched 97% of the time, and triage is 98% right on held-out data.

Challenges I ran into

The bug that only happened half the time. My first slow-phone profile slowed the CPU as well as the network. A timeout bug then reproduced only some of the time, which is useless as proof. Chrome's CPU throttling fired page timers late. Without it, the bug reproduced every time on a slow connection and never on a fast one.

A real app is not a demo app. On Actual Budget the first pass solved 0 of 8. The twin wandered while the app was still loading, pressed the same button nine times without noticing, and once "proved" a bug with an unrelated obstacle on the way. Each miss became a fix: wait until the app says it is ready, show the twin what each past tap hit, ask a second skeptic question for every piece of evidence, and refresh a page that never finishes loading. The latest run solved 1 strictly, and most misses now end with an honest "not reproduced" instead of a false proof.

The fixer looked in the wrong file. In a big repo the fixer only sees some files. For a date picker that would not open on Android, it read the transaction list, while the real bug sat in a shared input component one import away. Evidence now names the element (<input type="date">), and the file picker follows imports, including path aliases.

A free endpoint under load. Three benchmark jobs sharing one free key got "503 ResourceExhausted". Model calls now wait and retry, and the benchmark runs one job at a time.

Accomplishments that I'm proud of

  • The full chain runs on its own, from a complaint typed in the widget to a guarded pull request, on code I did not write.
  • The model cannot fake a result. "Reproduced" needs machine evidence; an unreadable model reply never decides anything; a fix counts only when code replays the original steps and the bug is gone.
  • I publish my misses. The real-bug table shows every case, including the ones I lost.

What I learned

  • Simulate conditions, don't role-play them. A model told "you are on a slow phone" finds nothing; a browser that really is slow finds the bug.
  • Separate acting from judging, and judge with evidence, not the model's opinion of its own work.
  • Real apps break agents in boring ways: loading screens, page reloads, alias imports, rate limits. A benchmark on real bugs found them in a day.

What's next

Built after the first draft, each tested on a planted bug end to end (reproduced, fixed, guarded): iPhone users in Safari's own engine, older users at their own pace (an OTP that expired before they finished typing), and a payment interrupted by a dropped signal (charged twice on retry). Next:

  1. Money and date formats. "1,00,000" against "100,000", month-end and year-end dates, time zones: the wrong amount charged, the wrong due date.
  2. Five design-partner teams with pull requests merged by people who are not me.
  3. Native Android apps, with an emulator in GitHub Actions.

Try it

Built With

Share this project:

Updates

Submission history