Inspiration

Every stream viewer has said it at least once: "Wait, find me that clip where he..."

You remember the moment perfectly. The red jersey, the scream, the exact line someone said. What you don't remember is the title, the date, or which of the 400 clips in the channel it lives in. Twitch search only knows titles, and titles are usually things like BoredHonestPhoneTebowing.

So I wanted a search box that works the way memory does. You describe what you saw and heard, and it takes you to the exact second.

What it does

Clipdexer turns clips into a searchable archive of moments.

Bring clips in by uploading a video or importing your own clips through the official Twitch API.
Every clip gets indexed once. Speech is transcribed with real timestamps, and a frame is sampled every 6 seconds and described by a vision model (clothing, objects, actions, scene).
Search like you remember. Try "wearing a red jersey and says my heart dropped". Clipdexer splits that into a visual clue and a speech clue, finds evidence for both close together, and drops you into the player at that timestamp.
It shows its work. Every result comes with the exact transcript lines and frame descriptions that matched, and you can click any of them to jump there. If nothing really matches, it says No confident match instead of making something up.

How I built it

The app is built with Next.js, React and TypeScript. FFmpeg handles the media work, and an OpenAI-compatible API provides the AI pieces: Whisper for transcription, a vision model for frame descriptions, and text-embedding-3-small for semantic search.

The indexing pipeline looks like this:

Validate the upload and create a playable H.264 preview
Extract audio and transcribe it into timestamped segments
Sample frames at $t = 0, 6, 12, \dots$ seconds and describe each one
Embed every segment and save the whole index

Search combines keyword matching with semantic similarity. For each clue $q$ and each saved segment $s$, I compute a keyword coverage $c$ (the share of clue words found in the segment) and the cosine similarity of their embeddings:

$$ \text{sim}(q, s) = \frac{\vec{q} \cdot \vec{s}}{\lVert \vec{q} \rVert \, \lVert \vec{s} \rVert} $$

Those are blended into a score for each clue:

$$ \text{score}(q, s) = 0.7\,c + 0.3\,\text{sim}(q, s) $$

A clip ranks highly when every clue finds supporting evidence inside the same 12-second window, because that's what "the moment where" really means. Results are labeled as strong, likely or partial matches, so you always know how much to trust them.

Twitch import uses the OAuth authorization-code flow and the official Get Clips Download endpoint. It doesn't scrape anything, and tokens never leave the server.

Challenges

Timestamps had to be real. It's easy to fake timestamps by spreading a transcript evenly across a video, but then "jump to the moment" lands in the wrong place. I only accept transcription output with actual segment timings, and I treat frame descriptions as what they are: snapshots of a single instant.
Calibrating semantic search. My first similarity thresholds looked reasonable on paper and almost never fired in practice. When I measured real queries against real clips, relevant pairs scored around $0.43$ to $0.55$, while unrelated text sat around $0.20$ to $0.35$. Retuning the cutoffs to fit that gap made paraphrased searches like "chick fil a commercial" suddenly work.
Understanding what people mean. "Wearing purple and talking about Bulgaria" contains two different kinds of clues. "Laughing, not the gaming clip" contains an exclusion. And "says 'not today'" contains a quote that should not be read as an exclusion. Parsing these without an LLM call on every search took a lot of careful rules and tests.
Being honest about weak matches. The hardest design decision was letting the app say "I'm not sure." A search tool that always shows a confident answer feels great right up until it's wrong.
Small, very real bugs. Windows antivirus briefly locking files during atomic saves, video byte-range requests for seeking, and Twitch requiring a manage-clips scope just to download your own clip all needed patient fixes.

What I learned

Retrieval quality depends far more on good evidence than on clever ranking math. A well-described frame beats a fancy formula.
Embedding similarities are relative to the model you use, so measure them instead of guessing.
Showing users why something matched builds more trust than a confidence percentage ever could.
Building with an AI coding agent is incredibly fast, but deciding what "correct" means is still on you.

What's next

Continuous channel sync, so new clips index themselves
A real vector database for big libraries
Searching across full VODs, not just clips
Sharing a link that opens straight to the moment

Good moments. Terrible memory. That's what Clipdexer is for.

The intro, title, scoring formula, thresholds and pipeline steps all come from your actual code and the measurements we took. Before submitting, check that the "What's next" items match your real plans and that the AI coding agent line in "What I learned" is how you want to describe your process.