ReplayForge

ReplayForge helps developers reproduce bugs, review a suggested fix, and verify that the problem is actually solved.

Comment

Inspiration

Bug reports usually arrive as fragments: a sentence, a screenshot, a console log, or a short recording. Turning that evidence into a trustworthy repair still requires a developer to reproduce the failure, review proposed writes, and prove that the fix actually works.

ReplayForge was built to make that engineering loop observable, auditable, and replayable.

What it does

ReplayForge accepts a concrete bug description and optional evidence, pins the source repository to a known commit, creates a disposable Git worktree, and asks Codex for a deterministic Playwright reproduction plan.

The workflow has two explicit human approval boundaries:

  1. Approval before writing the generated reproduction test.
  2. Approval before applying the proposed application patch.

A valid run preserves the failing test output, root-cause analysis, proposed patch, applied diff, approval records, passing verification output, regression results, screenshots, videos, traces, and a hashed artifact manifest.

This makes it possible to answer not only “Did the patch pass?” but also:

  • Did the reported failure actually reproduce?
  • Was the test written for the intended symptom?
  • Was the patch reviewed before it was applied?
  • Did the unchanged test pass afterward?
  • Can another reviewer replay the evidence?

How we built it

ReplayForge is a local-first TypeScript monorepo.

The web interface uses Next.js and React. The local orchestrator uses Node.js, typed contracts, SQLite with Drizzle, Playwright Chromium, and FFmpeg/ffprobe for local video evidence processing.

The system uses Git worktrees to isolate generated changes from the original branch. It validates repository paths, preserves raw command output, records artifact hashes, and keeps network access disabled by default.

Codex is connected through ChatGPT-managed authentication rather than an OpenAI API key. The recorded evidence bundle contains an authentic Codex-guided investigation using the runtime-reported GPT-5.6 Sol model with ultra reasoning effort.

Challenges we ran into

The difficult part was not generating a plausible patch. The difficult part was preventing an AI workflow from claiming success without verifiable evidence.

We had to solve several engineering challenges:

  • isolating generated changes from the original repository;
  • validating local paths and filesystem boundaries on Windows;
  • preserving raw process output and artifact metadata;
  • proving that the generated test failed for the reported reason;
  • requiring approval before test writes and patch writes;
  • binding approvals to the exact proposed payload;
  • preventing tests from being weakened to produce a green result;
  • keeping the browser verification flow deterministic;
  • producing an honest Recorded Evidence Mode without presenting prerecorded output as live.

Accomplishments that we're proud of

  • A real deterministic browser bug reproduced with Playwright.
  • Two separate human approval gates bound to exact payload hashes.
  • A failing reproduction test captured before patching.
  • The unchanged test passing after the approved patch.
  • Targeted and regression verification results preserved.
  • Before-and-after screenshots, videos, traces, and diffs available for replay.
  • A hashed artifact manifest for auditability.
  • A professional demo that clearly distinguishes Recorded Evidence Mode from Live Mode.
  • A complete local workflow that runs without an OpenAI API key.

What we learned

Reliable agentic developer tools need narrow scope, explicit write boundaries, and observable verification.

A generated fix is only useful when the original failure, the approved change, and the passing verification result are preserved together. Human approval is not just a safety feature; it is part of the engineering evidence.

We also learned that local developer tools must account for real operating-system behavior, including Windows process management, path validation, browser lifecycle, and filesystem permissions.

What's next for ReplayForge

The next phase is to support more frontend repositories and richer evidence intake. Planned improvements include GitHub Issue import, approved pull-request creation, reviewer collaboration, candidate patch comparison, CI evidence export, and a broader evaluation suite.

These extensions will remain subject to the same principles: explicit approval, isolated writes, reproducible tests, and verifiable evidence.

Built With

Share this project:

Updates