Match Cut

You know the video you want. You just can't name it. Match Cut searches over 7000 music videos by light, motion and cut rate, with an agent that watches alongside you and puts it on the screen.

Comment

Inspiration

Almost every agent product makes the agent an oracle behind a text box. You ask, it answers, and its output is the interface. The chat is the product.

I wanted to find out what happens when the agent is a peer on a surface that exists without it.

The other half came from a problem I have had for adjacent archive work I do in relationship to MTV REWIND (wantmymtv.org) and the RAD Archive (radarchive.org) . I keep an archive of music videos, and every entry carries a visual fingerprint: average motion, brightness, warmth, saturation, contrast, shot length, cut count. That data has been sitting there the whole time because there is no way to put it in a user interface. Nobody wants seven sliders. But you can hand it to an agent in one sentence: something slow, warm, and barely cut.

That is a search no music service can perform, and it is the kind of thing people actually want. You know the video you want. You just cannot name it.

What it does

Match Cut is an archive of over 7000 music videos spanning 1966 to 2026, covering 2,994 directors, with 49,753 hand written connections between entries. Every video has a visual fingerprint and a written cultural annotation.

The agent and you share one screen. Thirteen WebMCP tools are registered on the page:

  • search_archive and find_by_look, the second of which searches purely on visual character
  • follow_connection, which walks the graph and states the reason each edge exists
  • play, queue_set, and now_playing, so the agent controls the stage directly
  • get_annotation for the scholarship behind a video
  • keep, forget, my_taste, and more_like_my_taste, a collection you can both read and write
  • find_records, which turns a video you love into a record you can own
  • archive_stats

You can drive it yourself the whole time. Skip ahead, go back, keep something, and the agent re-plans out loud and keeps pace.

The economic layer

Agent discovery almost always dead ends at a recommendation. You get told about a thing and then you are on your own.

Match Cut closes that loop. When something lands, the agent calls find_records and the page draws a shelf of actual pressings: label, catalogue number, country, year, the lowest price currently listed, how many copies are for sale, and a link that goes straight to the marketplace listing. Vinyl, CD, cassette. Real inventory, live prices.

It matters more than a checkout button, because it puts something at stake. Taste stops being a feed you scroll and becomes a thing you can act on. That also makes it an honest model for an archive: the scholarship stays free, and the archive earns when it sends someone to a record they would never have found.

Three rules keep it from being a bolted on store. The agent may never invent a price or a release, and quotes only what comes back from the lookup. Buying happens on the marketplace, not here, so nobody is asked for payment details on a research tool. And a release whose price lookup hit the rate limit is still shown, marked as a listing rather than dropped, because a real record with an unknown price beats a shorter shelf.

How we built it

The corpus is 68MB of source annotation. Shipping that to a browser is not an option, so it is split three ways.

The boot payload is columnar and dictionary encoded: parallel arrays for id, artist, title, year, tier and the eight fingerprint values, with directors, narrative types and visual eras collapsed into lookup tables. That is 340KB gzipped, and the page is interactive with all 7,139 videos searchable in about 75ms.

The connection graph and tag data arrive in the background and merge in when they land.

The prose is the most valuable thing in the archive, so none of it is truncated. All 24MB lives in 128 shards addressed by an FNV-1a hash of the video id, computed identically in the Python builder and the browser, so a card knows which shard holds its own annotation without a lookup table.

Tools are registered through document.modelContext.registerTool. When no in-page agent is present the same tools are driven by a server-side conversation, so the experience degrades to a normal chat rather than to nothing.

Records come from the Discogs database through a small proxy, since Discogs sends no CORS headers and rate limits unauthenticated callers to 25 requests a minute. Results are deduplicated by master release so the shelf shows six different records rather than six pressings of one, physical formats are preferred over files, price lookups run sequentially with a gap to stay inside the limit, and everything is cached for twelve hours on the reasoning that a 1971 pressing does not change.

Challenges we ran into

A tool the interface cannot have. Agent tools are usually a strict subset of what buttons already do, which makes the agent a convenience layer. find_by_look cannot be a control at all, because the input is a sentence about how something feels to watch. That was the moment the project stopped being a chat wrapper.

Four bugs that made a large archive feel small. Identical prompts returned identical results, because ranking always returned the head of the list. Annotations were being cut mid sentence, and measurement showed 66 percent of the curatorial assessments and 61 percent of the director biographies were losing their endings. Sets looped forever instead of ending. The suggested prompts were hardcoded, so every session opened the same way. Individually these are small. Together they made 7,139 videos behave like about twelve.

The videos that do not move. Late in the build I noticed the archive occasionally serving something with a motion reading of zero and an average shot length of 222 seconds. That is a single unbroken frame for an entire song. It is album art with the audio over it, not a music video, and 461 entries were like that.

The obvious fix was to gate on low motion, and it was wrong. Rubber Johnny reads 1.6 on optical flow and has 331 cuts. It is dark, not static, and flow alone cannot tell those apart. Cut count is the real discriminator.

Replacing them beat deleting them, but a swapped video makes the stored fingerprint a lie, so I reverse engineered the original encoding by testing candidate formulas against entries whose values were already known. Brightness and saturation turned out to be raw 0 to 255 means, contrast the mean per frame standard deviation, motion Farneback optical flow scaled by 7.5. Warmth was the interesting one. Red over blue looked obvious and was wrong, since it implied the median video was twice as red as blue. Red plus green over blue fit almost exactly.

Trusting a run that was lying. The replacement job reported a 57 percent hit rate. Broken into blocks, one span of 40 consecutive cards had returned zero. That is not a property of an archive. Retrying three of them by hand returned three candidates each, so those were rate limiting, not missing videos. I threw out every miss, kept the successes, and rebuilt the job with pacing, a confirming second look on any empty result, and a backoff triggered by miss streaks.

Knowing when not to ship the fix. Of 174 verified replacements, 115 had annotations that analyse the stillness as the artifact. One reads that the static upload format is itself historically significant. Another treats 462 seconds and zero cuts as an ideal case study. Swapping in a cut heavy video would have left the panel describing something that was not on the screen. So those stay gated, and only 59 were merged. The scholarship is correct and the video simply is not one.

What we learned

The engineering was almost always right. The failures were in how the thing presented itself. Every bug above was small in isolation and changed the felt size of the archive enormously.

The more transferable lesson is about the relationship. Four things turned out to matter, and they are rare together:

A tool that exceeds the interface, so the agent extends the product instead of wrapping it. Survival under interruption, because most agent demos are scripts that break the moment you intervene. Citation instead of generation, since the agent quotes the archive and flags low confidence rather than synthesising confidently. And shared memory, which is the one I would bet on beyond this project. Agent memory is normally a private ledger the system keeps about you. Here the collection is a list you can open, edit, and play, and the agent reads exactly what you see.

What's next

Roughly 60,000 more videos have finished visual cataloguing and are waiting on cultural annotation. The visual layer alone is enough for find_by_look to work on them today, which would deepen the archive considerably before a single new word is written.

The static video work also continues. Every replacement gets checked against its own annotation before it merges, because a video that contradicts its own scholarship is worse than one that is missing.

Built With

Share this project:

Updates

Submission history