how it works

Watch with Codex

The same conversation, now present in the moment.

Inspiration

The difference between watching something with another person and merely discussing it later is timing.

It is the laugh at the exact frame, the quiet “wait, what did that mean?”, or the look you exchange when something unexpected happens.

AI can already summarize a video, answer questions about a transcript, or analyze a screenshot. But most of those experiences begin after the moment has passed. The model may understand the content, yet it does not share the timing, the interruption, or the viewer’s reaction.

I wanted to build the opposite: not another video summarizer and not a second chatbot embedded beside a player, but a way for the Codex conversation you already have to become present inside the same live experience.

What it does

Watch with Codex turns a video page into a shared timeline between a person and Codex.

The viewer pastes a YouTube link, direct video file, or embeddable player and asks Codex to watch along. The existing conversation remains the memory and response surface. There is no duplicate chat, separate account, or artificial companion persona.

While the video is playing, Codex keeps a WebMCP observation open. The viewer can choose a cadence of 5, 10, or 15 seconds of media time. Codex stays quiet by default and decides whether the current moment deserves a reaction, a short comment, an explanation, or no interruption at all.

Speech-heavy videos are already supported. For YouTube videos with captions, the active captions are captured together with the rendered player. At the five-second cadence, successive observations provide a rolling, time-aligned language context, while the existing conversation preserves what came before.

This lets Codex follow what is being said as well as what is visible. Lectures, tutorials, interviews, historical speeches, and captioned films are practical use cases today—not only videos with visually obvious action.

The viewer can also react with an emoji. That reaction is not a canned command. It immediately wakes the pending observation and arrives with the exact playback time and the current visible frame.

A 😂 might mean “this chart is ridiculous.” A ❓ might mean “what is the historical context of that sentence?” The meaning comes from the frame, the captions, the conversation, and the viewer’s intent—not from a hard-coded emoji-to-answer mapping.

Codex can answer inside the same conversation, publish one to five lightweight visual reactions directly over the player, or pause and resume supported videos when a longer explanation would compete with playback.

Why it matters

The core value is not constant AI commentary. It is reducing the distance between noticing something and getting useful help.

A student can tap ❓ in the middle of a difficult proof, let Codex pause the video, receive an explanation of that exact step, and continue without losing the place. A language learner can ask about a phrase while the caption, speaker, and visual context are still connected. Someone watching a historical speech can receive context at the sentence where it becomes relevant instead of searching for an explanation afterward.

The same interaction can also reduce physical and cognitive friction. A viewer who finds repeated pausing, scrubbing, switching tabs, or typing difficult can communicate with a single reaction while Codex handles the timing and playback. That can be useful for people with motor limitations, pain, fatigue, temporary injuries, or anyone who benefits from a lower-interaction interface.

In a conversation guided by a parent or teacher, younger learners can receive simpler explanations and deliberate pause points without navigating away from the lesson.

Because the existing Codex conversation owns the memory, the experience can adapt to the viewer’s actual goal: exam preparation, language learning, historical context, simplified explanations, accessibility support, or simply quiet companionship.

Voice will make that interaction even more natural, but the current one-tap workflow already removes much of the mechanical overhead between the viewer and the assistant.

Why WebMCP

Without WebMCP, a person and Codex can discuss the same video, but they cannot reliably inhabit the same moment.

A link has no live state. A screenshot is detached from playback. A transcript may contain the words, but it does not tell Codex where the viewer is now, whether they paused, rewound, changed playback speed, or reacted to a particular moment.

WebMCP is load-bearing in this project.

The page exposes five tools: watch_get_session, watch_observe_next_moment, watch_play, watch_pause, and watch_react.

The central tool, watch_observe_next_moment, acts as both a clock and an event channel. It resolves at the next scheduled media-time checkpoint or immediately when the viewer reacts.

Its cadence follows video playback time rather than wall-clock time, so pauses, playback-speed changes, and time spent reasoning do not silently move Codex farther away from the shared moment.

A persistent cursor preserves the schedule between observations, reports missed checkpoints, and resets cleanly after the viewer seeks to another part of the video.

WebMCP gives Codex the structured moment: the media identity, player state, exact timecode, viewer signal, and available actions. Browser observation then captures the rendered player immediately before reasoning, including cross-origin video, active captions, and visible overlays.

Together, these two layers let Codex know where the viewer is, see what the viewer sees, follow what is being said, and act inside the same experience.

Remove WebMCP, and Watch with Codex becomes an ordinary video player with chat beside it.

How I built it

Watch with Codex is a deliberately small Next.js and React application deployed on OpenAI Sites.

One page owns the live session: the selected source, normalized playback state, observation cadence, media-time cursor, pending viewer signals, and on-screen reactions. The existing Codex conversation owns memory, language, and interpretation. The project needs no database and no independent assistant backend.

YouTube playback uses the IFrame Player API, while title and author metadata are resolved through oEmbed. Direct video files use the browser’s native video element. Generic embeds remain viewable, but timing and playback tools fail explicitly when the page cannot access those capabilities.

One architectural boundary was especially important. A host page cannot read pixels from a cross-origin YouTube iframe. Instead of pretending otherwise or returning meaningless image data inside JSON, each WebMCP observation provides the exact checkpoint and a capture contract targeting .player-frame.

The browser captures the rendered frame immediately before Codex reasons about it. This also means that YouTube captions remain attached to the speaker, slide, chart, or scene they belong to rather than becoming a detached block of transcript text.

The observation loop has been tested across pauses, resumes, seeks, playback-speed changes, model latency, missed checkpoints, and viewer-triggered interruptions. The persistent media-time cursor keeps the session synchronized instead of restarting the clock after every model response or gradually drifting behind the video.

Deterministic code owns timing, validation, rate limits, and failure states. Codex owns the meaning of the moment.

Challenges

The hardest problem was not embedding a video or registering tools. It was making the interaction feel like company rather than surveillance or narration.

A companion that comments every five seconds quickly becomes unbearable. A companion that reacts several seconds too late no longer feels present. The observation loop therefore had to be quiet, event-driven, and anchored to media time.

Speech-heavy content created another important design requirement: the model needed linguistic context without turning the experience into a detached, post-video transcript workflow. For captioned YouTube videos, successive five-second observations preserve a rolling view of the spoken content inside the same live timeline. In testing, this has been enough for Codex to follow lectures and other dialogue-heavy material while remaining synchronized with the current visual moment.

Viewer reactions also needed to interrupt the pending observation immediately without stopping the watch-along. At the same time, model latency could not be allowed to restart the clock after every response and gradually move Codex behind the viewer.

The final challenge was respecting browser security boundaries. WebMCP can expose structured page state, but the page cannot directly inspect every rendered pixel inside a third-party player. Keeping the checkpoint and visual capture as two explicit, consecutive steps made the system both honest and useful.

Accomplishments I am proud of

I am proud that the result behaves like a small product rather than a technical demo with a tool panel.

A viewer can react without typing. Codex can understand why that reaction makes sense at that exact moment by combining the frame, active captions, playback state, conversation history, and viewer signal.

It can provide useful context, react back on the player, pause the shared video when an explanation needs space, and then continue watching without losing synchronization.

The same protocol supports more than one content category or one interaction style. It can move from a product presentation to a historical speech, lecture, tutorial, film, stream, or other time-based experience without changing the underlying architecture.

It can act as quiet company, a contextual tutor, or a lower-friction interface for someone who cannot comfortably control the video and type at the same time.

Most importantly, the page never decides what a human reaction means. It creates a precise shared moment and leaves interpretation to the model.

What I learned

Agent-native interfaces are not created by placing a chatbot next to an existing product.

They emerge when the product exposes the right state, timing, and actions while allowing the model to do what it is good at: understanding context and deciding how to respond.

I also learned that presence is partly restraint. The most human behavior is often not speaking. It is knowing when a small reaction is enough—and when the right moment deserves a real explanation.

What’s next

The next step is voice: letting the viewer ask a question aloud without leaving the video, allowing Codex to pause when necessary, answer naturally, and resume the experience.

YouTube captions already make lectures and other speech-heavy videos practical. Next, I want to add explicit transcript providers for direct files and other authorized media, live-caption support for streams, richer learning features such as saved explanations and bookmarks, and eventually shared sessions with multiple viewers.

But the central idea will remain the same:

Watch with Codex is not a video chatbot. WebMCP gives the existing assistant a shared timeline—the same conversation, now present in the moment.