lastmile

Inspiration

The edit is done and the video still isn't published. There's a title, a description, tags, chapter markers, a thumbnail and a posting time left. About twenty minutes, and it's the same twenty minutes every time.

The obvious fix is to paste the transcript into a language model and ask for all of it. I tried that first. It works for the words and fails for the numbers, because a model has no clock. It returns timestamps that look right and aren't, and a viewer who clicks a chapter lands in the wrong section.

So lastmile takes every number from the data and asks the model only for words.

What it does

Give it a finished video and its transcript. It returns chapters, a thumbnail, a title, a description, tags, and a scheduled upload.

Chapters come from word-level Whisper timings, the same numbers FFmpeg used to cut the video. The model names each section and never places one. Thumbnails are real frames graded by a vision model, not generated images, and every score is printed so you can see why a frame won.

How I built it

TypeScript owns control flow and every number. The model is asked for labels and titles only, and its reply is rejected if the label count disagrees with the boundary count. Wrong labels look authoritative in a way missing ones don't.

The vision scorer and the titler sit behind one-method interfaces. That started as a testability thing and turned out to matter more: simulating a rate limit or malformed JSON against a live API is impractical, so those paths usually go untested and break in production. Here they're three lines of stub, which is why 67 tests run with no API key and no network.

Challenges

The pause threshold that found nothing. The first version split chapters on silences longer than 900ms. That's the obvious approach and it works on raw footage. Against a real video it found nothing: largest gap 440ms, median 0. The clip had already been edited, and cutting dead air is what editing does, so there was no silence left to split on. The algorithm failed on exactly the videos people publish. It now ranks the gaps present in that specific transcript, so the threshold comes from the recording instead of an assumption about it. I kept the old behaviour as a passing test.

YouTube discards a malformed chapter list without telling you. Break one rule and the whole list is ignored. No error, no warning, the video just has no chapters, which is indistinguishable from never building the feature. Each rule is now code with a test, and if fewer than three chapters survive it emits none rather than a list that gets thrown away.

A test caught the dependency injection doing nothing. An early version had the frame scorer read the image file itself, so injecting a stub still hit the filesystem. The read moved into the dependency.

What I learned

Real data falsified my design on the first run. A fixed threshold that was obviously correct in theory found zero boundaries on an actual video. Now I test against a real 155-word transcript that ships as a fixture, and that fixture is what caught it.

Built With

• chromium
• ffmpeg
• ffprobe
• gpt-vision
• next.js
• node.js
• oauth2
• openai
• postgresql
• railway
• react
• remotion
• supabase
• typescript
• vercel
• vitest
• whisper
• youtube-data-api-v3