ReplayLab

ReplayLab turns captured user sessions into validated GPT-5.6 reproduction plans, then replays the exact experience and reproduces the bug safely in Playwright.

Comment

Inspiration

Customer support tickets often describe a bug, but they rarely contain enough context for a developer to reproduce it.

A user might say, “My coupon disappeared during checkout,” while the support team only receives a screenshot, a short message, and scattered logs. Developers then spend hours asking follow-up questions, rebuilding account state, searching through events, and guessing which action triggered the failure.

We built ReplayLab to turn that broken handoff into an executable reproduction workflow.

Instead of forwarding a vague ticket, ReplayLab captures what the user actually experienced, replays the session visually, asks GPT-5.6 to convert the evidence into a constrained reproduction plan, validates that plan, and safely runs it against a local test environment.

What it does

ReplayLab connects four steps that are normally separated:

  1. Capture the user session

    • A ReplayLab-enabled web app records DOM changes, user actions, semantic events, and selected application state.
    • Sensitive values such as passwords, tokens, cookies, authorization data, email addresses, and payment information are masked or excluded before export.

  2. Replay the exact experience

    • The captured session is exported as a strict .replaylab bundle.
    • The desktop app imports the bundle and provides visual session replay with play, pause, seek, speed controls, a synchronized event timeline, and state snapshots.
    • Developers can see the exact moment an expected state becomes an incorrect state.

  3. Generate a reproduction plan with GPT-5.6

    • GPT-5.6 receives the sanitized ticket, environment, events, and snapshots.
    • It produces a strict Structured Output containing hypotheses, allowlisted reproduction actions, and expected assertions.
    • The model cannot provide arbitrary URLs, selectors, scripts, shell commands, or executable code.
    • Every generated plan is parsed with Zod and checked by an additional semantic validator before it can run.

  4. Reproduce the bug safely

    • A deterministic dry-run evaluates the plan without launching a browser.
    • A fixed Playwright adapter then runs the validated actions in a local synthetic fixture.
    • ReplayLab compares the original captured session with the automated reproduction result.

Our demo reproduces a checkout issue called BUG-1042:

  • The user applies the SAVE20 coupon.
  • The discount becomes 20,000 KRW.
  • The shipping country changes from South Korea to Japan.
  • The user opens checkout.
  • The coupon unexpectedly changes from APPLIED to REMOVED.
  • The discount changes from 20,000 KRW to 0.

ReplayLab captures that transition, highlights the first anomalous moment, and reproduces the same failure through Playwright.

How we built it

ReplayLab is an Electron desktop application built with React and TypeScript.

The system is divided into several security boundaries:

Capture layer

The synthetic Demo Store uses rrweb to record actual DOM events. We also record semantic actions and selected state snapshots, including:

  • Product and cart state
  • Coupon status
  • Discount amount
  • Shipping country
  • Address state
  • Checkout state

The capture remains in browser memory until the user explicitly reports the issue. It is then exported locally as a strict .replaylab JSON bundle.

Case workspace

Imported sessions become persistent cases inside ReplayLab.

Each case includes:

  • Overview
  • Evidence
  • Event timeline
  • Snapshot diff
  • Session replay
  • Reproduction plan
  • Automated reproduction result
  • Export workspace

Snapshot comparison supports added, removed, changed, and type-changed values while blocking prototype keys and redacting sensitive paths.

GPT-5.6 planning layer

GPT-5.6 is used for the part that requires reasoning rather than deterministic code: transforming incomplete user evidence into a minimal reproduction hypothesis and structured action plan.

We use:

  • OpenAI Responses API
  • Strict Structured Outputs
  • A fixed GPT-5.6 model policy
  • store: false
  • No tools
  • No web access
  • No arbitrary code generation
  • A maximum of ten allowlisted actions

The output is never trusted directly. It must pass:

  1. Strict transport schema validation
  2. Existing Zod domain validation
  3. Case ID validation
  4. Hypothesis consistency validation
  5. Action ordering validation
  6. Privacy and secret checks
  7. Semantic validation against the captured case

Only validated plans are stored in an in-memory registry and made available to the execution engine.

Reproduction layer

ReplayLab contains two execution paths:

  • A deterministic in-memory dry-run
  • A real Playwright Chromium run against a local synthetic fixture

The Playwright adapter uses fixed internal test IDs. The renderer and GPT-5.6 cannot provide a selector, URL, executable path, browser argument, shell command, or script.

External browser requests are blocked. Browser, context, page, and fixture server resources are closed after every run.

Local BYOK security

Users provide their own OpenAI API key through the desktop app.

The key:

  • Is handled only by the Electron Main process
  • Is encrypted with Electron safeStorage
  • Is stored under the application user-data directory
  • Is never returned to the renderer
  • Is never displayed after storage
  • Is never committed to Git
  • Is not included in captured sessions or exported cases

API usage may incur charges on the user’s own OpenAI account.

Challenges we faced

Replaying what the user actually saw

A screenshot was not enough. We needed to preserve the relationship between user actions, DOM changes, and internal state.

We solved this by combining rrweb visual recording with semantic events and explicit application snapshots. This makes it possible to replay the interface while also explaining what changed underneath it.

Preventing AI output from becoming arbitrary execution

Letting a model generate Playwright code would have been powerful but unsafe.

Instead, GPT-5.6 produces a small domain-specific plan. Every action belongs to a predefined allowlist and is mapped to a fixed adapter written by us. The model proposes intent, while deterministic code controls execution.

Distinguishing reproduction from ordinary failure

An assertion failure does not automatically mean the original bug was reproduced.

For BUG-1042, ReplayLab only reports reproduced: true when the complete signature matches:

  • The coupon was previously applied
  • The country changed from KR to JP
  • The address was saved
  • Checkout opened
  • The coupon became removed
  • The discount became zero
  • The expected state remained applied with a 20,000 KRW discount

Browser crashes, missing elements, invalid plans, and unrelated assertion failures are reported separately.

Privacy

Session recording can easily collect too much information.

ReplayLab applies masking and redaction before export, limits capture size and duration, excludes request and response bodies, blocks cookies and authorization data, and validates every imported bundle before saving it.

What we learned

We learned that AI is most valuable here when it is placed between evidence and deterministic execution.

GPT-5.6 should not control the browser directly. Its role is to interpret noisy evidence, identify a likely minimal sequence, and express that reasoning through a constrained schema.

We also learned that visual replay and automated reproduction solve different problems:

  • Session replay shows what the user experienced.
  • Automated reproduction proves whether the same failure can be triggered again.
  • GPT-5.6 connects those two worlds by converting evidence into a validated plan.

How we used Codex

Codex was used throughout the full development lifecycle:

  • Architecture and security-boundary design
  • Electron, React, and TypeScript implementation
  • Strict schemas and semantic validators
  • rrweb capture and replay integration
  • Playwright fixture and adapter implementation
  • Test generation and debugging
  • Git branching, commits, and pull requests
  • Documentation, release preparation, and demo planning

The project was developed as a sequence of narrow tickets. Each ticket required tests, linting, type checking, production builds, security checks, and a structured handoff before moving to the next stage.

Accomplishments

  • Actual DOM-based user-session capture
  • Downloadable and importable .replaylab case bundles
  • Visual replay synchronized with actions and state
  • Sensitive-value masking and strict bundle validation
  • GPT-5.6 Structured Output planning boundary
  • Deterministic reproduction dry-run
  • Real local Chromium reproduction with Playwright
  • Fixed external-network and execution boundaries
  • Persistent Electron case workspace
  • More than 190 automated unit, security, replay, browser, and integration tests

What is next

Future versions of ReplayLab could add:

  • An installable Capture SDK for third-party web applications
  • A consent-based rolling buffer for the last few minutes of activity
  • Direct integration with customer-support systems
  • Sanitized trace and regression-test export
  • Team collaboration and case assignment
  • Additional verified adapters for different product environments

The current MVP intentionally supports a controlled Demo Store environment. It proves the complete workflow without claiming support for arbitrary production websites.

Built With

Share this project:

Updates