ReplayOps — turning a screen recording into something a team can actually reuse

Why we started building it

ReplayOps came from a problem we have seen during knowledge-transfer sessions.

A senior employee shares their screen, completes a task, explains a few important decisions, and answers questions. The session gets recorded, but a week later the new employee still has to ask:

Where exactly did you click?
Why did you choose that option?
What happens if this field is missing?
Does this step need approval?
Can I repeat the same process with a different input?

The answer is usually hidden somewhere inside a long recording, an incomplete document, or the memory of the person who conducted the session.

We initially thought about building a better recording and SOP tool. While testing the idea, we realized that documentation alone was not enough. If ReplayOps could understand the demonstrated process, it should also make that knowledge searchable and help repeat the safe parts of the workflow.

That is how the project became:

Record the work, preserve the reasoning, ask questions about it, and safely replay the repeatable steps.

What we built

ReplayOps is a browser-based operational-memory and workflow-replay product.

A user records a browser task while optionally explaining what they are doing. ReplayOps captures the screen evidence, browser actions, typed inputs, page context, navigation, screenshots, audio, and tab changes.

After the recording, GPT-5.6 turns that evidence into a reusable workflow containing:

A structured SOP
Business rules and decisions
Preconditions and expected outcomes
Exceptions and unanswered questions
Annotated visual evidence
Searchable knowledge for cited Q&A
A reviewed browser replay plan

The result is stored in a local workflow library. Selecting a workflow also selects the corresponding knowledge, SOP, chat context, evidence, replay plan, and audit history.

The product flow

Record

ReplayOps can record from the website or from its Chrome/Brave extension.

The extension adds a floating assistant directly to ordinary webpages. It records meaningful browser events such as navigation, clicks, completed input values, selections, scrolling, media actions, and multi-tab transitions.

Microphone narration is optional, but it adds useful context. For example, the screen may show that the user selected a queue, while the narration explains why that queue was correct.

The recorder also captures readable page context and representative screenshots. We intentionally do not treat every mouse movement as a meaningful workflow step.

Build knowledge

When the user selects Build knowledge, GPT-5.6 receives a structured evidence package rather than only a video.

That package can include:

Browser actions
Page titles and URLs
Readable page text
Typed values
Narration transcript
Screenshots
Timing
Tab boundaries
Recorded selector information

GPT-5.6 aligns this evidence and produces structured operational memory using validated schemas.

One important rule in our design is that ReplayOps must distinguish between:

Something the user demonstrated
Something the user said
Something that was visible on the page
Something inferred by the model

If information was not captured, the generated memory can mark it as unresolved instead of quietly inventing an answer.

Review the SOP

ReplayOps creates an evidence-linked SOP that can be previewed before export.

The SOP includes the purpose, audience, procedure, decisions, exceptions, and visual evidence. Captured screenshots can be annotated to highlight the area involved in an interaction.

The document can be exported as:

Word (.docx)
Markdown
Confluence-compatible HTML

We also built a controlled enterprise-delivery demonstration. It shows how an approved SOP could pass through a policy gate and delivery adapter without requiring us to pretend that a real Microsoft tenant is connected.

Ask questions

The Chat workspace lets the user ask questions about the selected recording without relying on prewritten prompts.

ReplayOps retrieves relevant parts of the operational memory and produces a cited answer.

For example:

“Why was this option selected?”
“Which actions require approval?”
“What did the presenter say about this exception?”
“What should a new employee verify before starting?”
“Was this value typed during the recording or already present?”

If the workflow memory does not contain enough evidence, the assistant should say that rather than answer from unrelated general knowledge.

Replay

Replay was the most difficult and most educational part of the project.

We did not want ReplayOps to be a macro that blindly clicks stored screen coordinates. Modern sites rerender elements, generate dynamic IDs, move content, and frequently present multiple elements with similar text.

Our replay engine therefore resolves a target using several signals:

Stable attributes
Accessible role and name
Visible text
Nearby and ancestor context
Recorded screen region
Structural information
Coordinates as a final fallback

Before performing an action, ReplayOps waits for the page to settle and checks that the target is visible, stable, and not obstructed. After the action, it verifies whether the expected result occurred.

If a target cannot be identified safely, ReplayOps pauses. The user can retry, select the correct element, skip an explicitly reversible step, restart from step one, or stop the replay.

This was a deliberate product decision. We would rather demonstrate an honest pause than show an agent confidently clicking the wrong job listing, button, or account action.

Library and workflow packets

The hackathon build stores up to eight workflows locally in the browser.

Important workflows can also be downloaded as .replayops.json packets and imported later. A packet contains the structured memory, SOP information, evidence metadata, and replay instructions.

It does not contain passwords, browser cookies, login sessions, API keys, or raw video.

This gives us the beginning of a workflow-sharing format that a team could later review, version, approve, and distribute.

Trust and audit history

ReplayOps includes a Trust workspace because recording and automation need more than a success message.

It shows information such as:

Evidence coverage
Replay readiness
Grounded-memory coverage
Source composition
Approval requirements
Replay retries and pauses
Human-selected targets
Execution outcomes

The purpose is to make the system inspectable, especially when it decides not to continue automatically.

How we built it

ReplayOps is a TypeScript monorepo.

The website uses Next.js and React. Shared Zod schemas validate the knowledge, SOP, replay, packet, and audit contracts.

The browser extension uses Manifest V3 and includes:

A Shadow DOM floating assistant
Content scripts for action and page-context capture
An offscreen document for persistent MediaRecorder capture
Service-worker coordination
Multi-tab workflow tracking
Local extension storage and IndexedDB
A browser replay coordinator

We use the OpenAI Responses API with GPT-5.6 for evidence alignment, operational-memory generation, replay planning, and grounded Q&A. Optional narration is transcribed using an OpenAI transcription model.

The model interprets the captured work, but it does not directly click the webpage. The deterministic extension handles execution, verification, retries, approvals, and stopping.

That separation became one of the most important architectural decisions in the project.

How Codex helped us build it

Codex was our main engineering collaborator during Build Week.

This was not a case where we generated the initial project once and stopped. We used the same Codex task throughout repeated implementation, testing, failure analysis, and redesign.

For example, Codex helped us:

Structure the monorepo and shared schemas
Build the initial Next.js product
Build and repeatedly refactor the Manifest V3 extension
Move recording into an offscreen document so navigation would not immediately destroy it
Track workflows across browser tabs
Diagnose service-worker suspension and closed message ports
Improve replay target scoring
Add page-settle waits and retry polling
Implement post-action verification
Debug IndexedDB and browser-local persistence
Generate annotated SOP views and Word exports
Add workflow packets and validation
Build the Trust workspace and replay audit events
Write regression tests for bugs we encountered
Run type checking, linting, tests, extension validation, and production builds

The key product choices still came from us.

We decided to keep the hackathon version local-first. We decided that packets must exclude authentication data. We decided to separate GPT-5.6 reasoning from deterministic browser execution. Most importantly, we decided that replay should pause when uncertain instead of guessing.

Our main Codex session ID is:

019f7018-dd0a-7033-9cd6-9af5d42a6f80

What went wrong along the way

The extension caused most of our difficult bugs.

At different stages, the floating assistant disappeared after navigation, recording stopped during refresh, service-worker message channels closed, screen capture required confusing extra interactions, and browser storage failed while saving larger evidence.

Replay also exposed problems that were easy to miss in a simple prototype.

A target that looked obvious to us could have several similar DOM candidates. LinkedIn-style dynamic lists were especially challenging because job titles repeat and the contents move. Media players often expose different controls depending on whether they are paused, playing, or covered by another element. Some sites use controlled React inputs that do not respond correctly to a simple synthetic value assignment.

We worked through these failures by adding:

Stable logical tab identities
Persistent replay checkpoints
Multi-signal target scoring
Visibility and obstruction checks
Page-settle detection
Bounded retry polling
Controlled-input handling
Post-action verification
Human-assisted target healing

Replay is much stronger than our first version, but we are also honest about its current boundary. It is designed for reviewed browser workflows, not unrestricted autonomous browsing.

What we learned

The biggest lesson was that understanding a process and executing a process are different technical problems.

GPT-5.6 is good at combining narration, screenshots, page context, and browser actions into something meaningful. It can identify rules and create a much more useful SOP than a raw click list.

Execution needs a different mindset. It must be deterministic, observable, interruptible, and verified step by step.

We also learned that a safe pause is not necessarily a failed agent. If several visible buttons look equally plausible, asking the user is better than silently choosing one and recording the wrong outcome as success.

Finally, we learned that the most valuable part of a recorded workflow is often not the click itself. It is the explanation, exception, or decision surrounding that click.

Current scope

For this hackathon, ReplayOps focuses on browser workflows in Chrome and Brave.

It does not claim to automate native Excel, Teams, Zoom, ServiceNow desktop clients, or arbitrary desktop applications. Those applications may appear in a screen recording as visual evidence, but the current replay engine operates on supported webpages.

The current version also uses browser-local storage and retains a maximum of eight workflows. A production enterprise version would require authentication, encrypted tenant storage, redaction policies, retention controls, and organization-level permissions.

We chose to make these limitations visible rather than hide them behind a polished demo.

What we want to build next

Our next major step is a native desktop companion.

That would allow ReplayOps to extend the same consent, evidence, approval, and audit model to meeting tools, spreadsheets, and desktop enterprise applications.

Other planned improvements include: