Keyframe

Teach GPT-5.6 by showing it what to do. Keyframe turns videos into timestamped visual and spoken evidence that Codex can use to build, verify, and explain its work.

Comment

Inspiration

I was inspired by all the MCP servers and plugins that exist for ChatGPT and Codex, as well as just my own personal experience on how AI really doesn't know how to watch videos. This gives it the ability to do that.

What it does

Keyframe is a local MCP server and plugin that gives Codex, ChatGPT, and other AI agents the ability to understand both what videos say and what they visually show.

Instead of describing a complex workflow through a long prompt, a developer can record themselves performing the task. Keyframe indexes the explanation, on-screen actions, visible code, OCR text, and important frames. GPT-5.6 can then retrieve the relevant evidence, apply the demonstrated approach in a target repository, run tests, and cite the timestamps that informed its decisions.

Keyframe supports uploaded videos, public YouTube and Loom videos, animated GIFs, and tutorial discovery through the host agent’s web search. It is designed for developer demonstrations, bug reproductions, internal walkthroughs, tutorials, and any workflow where showing an agent is easier than prompting it.

How we built it

I built Keyframe primarily through a Codex session using GPT-5.6. I began with a detailed specification, then used Codex to plan and implement the ingestion pipeline, MCP tools, local indexing, OCR, frame extraction, retrieval, plugin integration, tests, and documentation.

The architecture intentionally separates deterministic video processing from model reasoning. Keyframe handles acquisition, transcription, OCR, indexing, timestamp retrieval, and source frames locally. GPT-5.6 is then responsible for interpreting that evidence, applying it to the user’s task, modifying code, and running tests.

Most of the development process involved iterative testing with uploaded videos, YouTube videos, Loom recordings, and generated fixtures. Codex helped identify bottlenecks, improve processing speed, harden the retrieval workflow, reduce unsupported claims, and create reproducible end-to-end tests for judges and users.

Challenges we ran into

The two hardest problems were processing speed and grounding.

Early versions took too long to analyze short recordings, so I worked with Codex to reduce unnecessary frame extraction, add bounded retrieval, improve caching, and separate fast analysis from deeper visual inspection.

The second challenge was preventing agents from making claims about video content they had not actually verified. I refined the MCP instructions and retrieval workflow so agents retrieve narrow timestamped evidence, inspect source frames when needed, distinguish transcript evidence from visual evidence, and avoid claiming that something appeared unless it was supported by the indexed media.

Accomplishments that we're proud of

I'm proud that I was able to get a working version of Keyframe and be able to use it to in just some videos that I wanted AI to look at and help build some projects. The fact that I'm actually able to use it already. It's very impressive. And I'm going to keep using it in the future with Codex.

What we learned

I learned that we have a long way to go on enhancing agentic AI capabilities. Keyframe extends Frontier AI by giving it the ability to watch videos on the go. This can save developers hundreds of hours on their workflows.

What's next for Keyframe

Keyframe will remain an open source project so that developers all around the world can use it to be able to enhance their agentic AI building experience.

Built With

Share this project:

Updates