ReproFix - Autonomous Issue Reproduction

ReproFix turns plain-language bug reports into reproducible failures: an autonomous browser agent reproduces issues, captures evidence, files Jira tickets, and replies to reporters.

Comment

Inspiration

Every engineering team knows the ritual: a bug report arrives saying "checkout is broken," and a product specialist/support spend hours trying to reproduce the issue to determine whether this is a bug, a cloud/infra issue, or if it is expected behavior. Then if needed, they would also need to spend time creating jiras for countless of issue reported to forward it to the right team to take care of.

Developer spends the next hour interpreting vague request, manually replaying steps, staring at the network tab, guessing at the first place things went wrong, writing up findings.... then doing it all again after the fix to confirm.

The report-to-reproduction gap is the most repetitive, least automatable-feeling part of the support loop. We wanted to close it: let an agent do the browser work, but keep every verdict anchored in captured evidence rather than model confidence.

What it does

ReproFix turns a plain-language bug report into a reproducible failure — and closes the support loop around it.

A reporter files a ticket in their own words on a support portal. Behind the scenes, ReproFix compiles the report into a typed workflow, executes it in a real browser session, captures network/console/DOM evidence, pinpoints the first step where expected and observed behavior diverge, and produces an evidence-grounded diagnosis.

From there, it routes the outcome: product bugs file a Jira issue on the client's dev board, infrastructure failures file to the DevOps board, and the reporter gets a plain-language reply — "our engineering team is investigating — all without a human touching the loop.

When a fix ships, the same stored workflow reruns to verify it deterministically: fix_verified / still_failing.

How we built it

  • Engine: Python + FastAPI. The report compiles into a typed WorkflowSpec IR (Pydantic) — semantic intents, parameters, and postconditions — not a generated script.
  • Compiler: Heuristic-first for speed; vague reports escalate to an LLM (OpenRouter) with sanitization at the IR boundary. The model can propose steps, but only non-inventable postconditions survive.
  • Runtime: An explicit state machine — observe → plan → resolve → execute → validate per step — with bounded retries, loop detection, and mid-run auth-wall recovery. If the app bounces to a login page the workflow didn't plan for, the agent signs in with the test account and continues.
  • Browser: Playwright, with Steel.dev cloud sessions for remote targets. The live session viewer is embedded in the dashboard.
  • Diagnosis: Rule-based classification over typed evidence for backend, frontend, performance, environment, and authentication failures. An LLM may write the narrative, but it can never claim more than the evidence shows.
  • Triage: Category → Jira project mapping via the Jira REST API, plus reporter-safe responses through a /ticket endpoint that exposes no internals.
  • UI: Next.js, with an internal engineering dashboard for live SSE traces, evidence, and verdicts, plus a separate reporter portal.
  • DemoShop: A controlled e-commerce target with 14 injectable faults — API 500s, hangs, crashes, and UI mutations — so every verdict class is demonstrable.

Challenges we ran into

  • Trusting LLM output at the boundary. The model invented postconditions like "expect text 'Demo Product'," used incorrect parameter names such as coupon_code instead of coupon, and emitted hallucinated step types — each producing false divergences. We built a sanitization layer that normalizes aliases, maps novel intents to canonical handlers, and discards inventable assertions.

  • Separating "server answers wrongly" from "server gone." A 503 and a connection refusal can look identical to a user. Our evidence model distinguishes them into product_bug vs. infrastructure and routes them accordingly.

Accomplishments that we're proud of

  • The verdicts are evidence-locked. Rule-based classification reads only captured signals; the LLM narrative cannot override them. Nothing is claimed beyond what the browser observed.

  • A genuinely closed loop. One shopper report → reproduced failure → first divergence → Jira ticket on the correct team's board → human-readable reply. Verified live end-to-end.

  • Honest degradation. Under-specified reports, missing features, and dead targets produce inconclusive / not_reproduced verdicts rather than fabricated bugs. The system never invents a failure.

  • 55 deterministic tests covering compiler, IR, resolver, redaction, diagnosis, and triage — with no browser or LLM required.

What we learned

  • Constrain the model, don't trust it. LLMs are excellent at semantic interpretation — such as understanding "smash the place order button" — but unreliable at assertions. The model proposes what to do, while typed postconditions decide what happened.

  • "Not a bug" is a verdict worth defending. Expected-behavior and inconclusive outcomes are first-class results; a system that always finds a bug isn't useful.

  • Vague reports need honest answers, not heroic guesses. When a report names no product and no login state, the right output is "couldn't exercise the flow" — with the specific reason instead of reproducing the issue blindly.

What's next for ReproFix — Autonomous Issue Reproduction

  • Deeper diagnosis: Correlating browser evidence with target-side telemetry such as server logs and traces to point at a code path, not just an endpoint.

  • The PATCH stage: Proposing the fix itself, closing REPORT → REPRODUCE → DIAGNOSE → PATCH → VERIFY.

  • Ticket status sync: Pulling Jira status back into the portal so reporters can see resolution progress.

Built With

Share this project:

Updates