Inspiration
Bugs usually get reported as a screen recording, a screenshot, or a quick description. The recording shows what went wrong, but that’s where the real work starts. A developer still has to reproduce the bug, figure out what happened, track down the code responsible, write a test, make a fix, and check that nothing else broke.
We wanted to close that gap between seeing a bug and fixing it.
Multimodal AI and specialized agents made us think this was possible. What if a screen recording could kick off an actual engineering workflow? Instead of just describing the bug, the system would understand what happened, dig through the codebase, reproduce the problem, and propose a fix backed by test results. That’s how Replay started.
What it does
Replay turns a screen recording of a bug into a reproducible test, a targeted code patch, and a verification report.
You can pick a sample recording or upload your own video. Then Replay runs it through a multi-agent pipeline:
Ingest: FFmpeg pulls timestamped frames from the video. Watcher: A vision-capable model looks at the UI, spots what’s off, and writes a structured bug report. Locator: The agent searches the codebase for the files, functions, and code most likely involved. Reproducer: It writes a focused Playwright test that should trigger the bug. Verifier: It checks that the test reproduces the same problem shown in the recording. Fixer: It writes a minimal patch aimed at the suspected issue. Reporter: It produces a report covering the analysis, the code changes, and the evidence.
Replay doesn’t take the model’s word for whether a fix works. It relies on real test results and deterministic checks to tell apart a bug that was actually fixed, one that was only reproduced, one that failed, and one that couldn’t run at all.
The workspace shows the video, evidence frames, agent activity, suspect code, generated tests, diffs, terminal output, and the final report.
How we built it
Replay is a TypeScript monorepo. The frontend is Next.js and React, the API is built on Hono, and background jobs run through a persistent worker. The AI agents live in their own packages.
Models: We connected an OpenAI-compatible client to Nebius Token Factory and set up role-based model configs for NVIDIA Nemotron models. Each stage uses a model suited to its job: vision for the video, code localization, test and patch generation, and reporting.
Video: FFmpeg extracts frames from WebM and MP4 files. The Watcher works from these frames, so it’s looking at what actually appeared on screen, not just a text summary of the bug.
Orchestration: Six agents handle video understanding, code localization, reproduction, verification, fixing, and reporting. Zod schemas validate the structured model outputs, and typed pipeline events show what’s happening at each step.
Browser testing: Playwright runs the reproduction tests against the target app. Deterministic gates confirm that the original bug reproduces, that the same symptom goes away after the patch, and that the regression suite still passes.
Execution and storage: A job queue and worker process handle the work, with PostgreSQL through Drizzle ORM for data. Model-generated code and browser runs happen in Docker sandboxes with resource limits, restricted networking, and filtered environment variables.
Frontend: The live run view includes the video player, evidence panel, agent timeline, code and diff views, terminal output, a benchmark dashboard, and generated reports.
We also built a Demo Shop app with five planted bugs and recordings of each, so we could test the pipeline the same way every time.
Challenges we ran into
The hardest part was connecting what the video shows to the code that causes it. A video might show a wrong total, a broken checkout, or a layout change, but finding the cause takes more than image recognition. The Watcher and Locator had to pass useful, structured evidence back and forth.
We also had to make sure the generated test failed for the right reason. A test can fail on a bad selector or a timeout and look like a reproduction when it isn’t. We added verification gates to catch unrelated failures and unexpected passes.
Trusting AI-generated fixes was another problem. A model can write a patch that looks right and doesn’t fix anything. So a fix only counts as FIXED after the bug stops reproducing and the regression tests pass.
Security mattered a lot too, since Replay runs generated tests and patches. We used isolated Docker execution, non-root processes, resource limits, restricted networking, protected paths, and secret filtering.
Deployment was tricky as well. Vercel works well for the web app, but background workers, durable storage, and isolated browser runs need separate infrastructure. We also had to be clear about what was a recorded demo and what was a live AI run, so we never presented precomputed results as fresh analysis.
Accomplishments we’re proud of A full multi-agent pipeline that goes from bug video to analysis, localization, test generation, verification, patch, and report. Vision-based video analysis working together with code-focused agents. Five reproducible Demo Shop bugs: cart-nan, double-discount, string-sort, empty-email, and quantity-reset. In our benchmarks, 5/5 bugs reproduced and 5/5 had the correct top-ranked code location. In our latest local live end-to-end run, all five cases reached FIXED. Verification gates that won’t mark a patch successful just because the model says so. Real-time events, run history, evidence panels, diffs, and detailed Markdown reports. Sandbox controls for model-generated code and Playwright runs. A responsive interface with sample recordings, custom uploads, a benchmark dashboard, and an interactive assistant. A public deployment on Vercel with production checks for the recorded sample replay.
The part we’re most proud of is that verification is built into the workflow itself. We didn’t want Replay to just suggest a fix. We wanted it to show whether the fix works.
What we learned
Multimodal AI gets much more useful once visual understanding is tied to real dev tools and repo context. A vision model can tell you what looks wrong on screen, but it can’t tell you which line to change. That takes code localization, reproducible tests, deterministic verification, and clear communication between agents.
We also learned to treat AI-generated code as untrusted. Isolation, resource limits, secret protection, and strict checks are requirements, not extras.
A model giving a good answer and a bug actually being fixed are two different things. Reliable AI engineering needs measurable evidence, honest failure states, and a clear record of what ran and what didn’t.
Finally, infrastructure is part of the product. Job queues, database consistency, worker uptime, and sandboxing matter as much as model quality.
What’s next for Replay
We want to move Replay past the benchmark app and make it a production-ready debugging tool. Next up:
Live execution in production: Connect the Vercel frontend and API to durable PostgreSQL, a persistent worker, secure isolated browser runs, and real model credentials, so every live request does genuine analysis. Better video understanding: Improve temporal reasoning, frame selection, recognition of UI interactions, and grounding bug detection in the evidence. Support for more apps: Go beyond the Demo Shop to arbitrary repos, frameworks, and real web apps. Better fixes: Improve reproduction accuracy, root-cause localization, patch quality, and regression prevention, using larger evaluation datasets. Developer integrations: GitHub pull requests, CI/CD, issue trackers, and automatic regression-test generation. Collaboration: User workspaces, persistent project history, secure artifact sharing, and team debugging workflows.
Our long-term goal is simple. Developers should be able to show Replay what went wrong, and Replay should help figure out why it happened, where to fix it, and whether the fix really works.
Replay: Show the bug. Get the fix. Prove it works.
Note: I worked from the text you pasted and didn’t open the GitHub repo, so the rewrite only reflects what’s in the write-up.
read the repo
I read the repo’s main page. It’s a public repo with a README, a targets/demo-shop app, and a packages/ and apps/ layout. I only saw the README and the file list, not the source code, so I can’t confirm how the agents are actually implemented.
The README also says something important that the write-up didn’t: the numbers are more mixed than the write-up suggests. The README’s benchmark table lists three of five bugs as FIXED (cart-nan, empty-email, quantity-reset), and two as REPRODUCED_ONLY (double-discount, string-sort). Its aggregate line says 5/5 reproduced, 5/5 located, and 3/5 fixed end-to-end. The write-up claimed all five reached FIXED, so I’d fix that before submitting. The README also describes the live app on Vercel as a recorded-sample replay unless a database, model key, and worker are configured, so the write-up’s “public deployment” claim needs the same qualifier.
Here’s the rewritten “What it does” section with those corrections:
What it does
Replay turns a screen recording of a bug into a reproducible test, a targeted code patch, and a verification report.
You can pick a sample recording or upload your own video. Replay then runs it through a seven-stage pipeline:
Ingest: FFmpeg pulls timestamped frames from the video. Watch: A vision model looks at the frames and writes a bug report with expected and actual behavior. Locate: The agent searches the codebase for the most likely buggy files and lines. Reproduce: It writes a Playwright test that should trigger the bug. Verify: It checks that the test reproduces the same problem shown in the recording. Fix: It writes a minimal patch for the suspected issue. Report: It produces a Markdown report with the analysis, changes, and evidence.
Replay doesn’t rely on the model saying a fix works. Deterministic gates use real test results to separate a bug that was fixed from one that was only reproduced, one that failed, and one that couldn’t run.
