Hindsight

A retention linter for video editors. Learns where your audience left your published videos, then flags the same problems in a rough cut before you upload it.

Comment

Inspiration

Every YouTube automation tool works downstream of the export — thumbnails, titles, scheduling. All of it comes after the decision that actually determines whether a video works: the edit.

Creators do get retention data, but too late to matter. By the time you see viewers leaving at 2:14, it's published.

Your published videos already record what loses your audience. That record should warn you about your next video, while it's still a rough cut.

What it does

Hindsight is a linter for video edits.

Point it at a rough cut and it runs ffmpeg, extracting shot boundaries and a loudness envelope to flag pacing outliers. On a real 2:57 video: a 129s unbroken shot, 4.3x the median, flat audio throughout.

Connect a YouTube channel and it pulls real retention curves, locates statistically significant abandonment points, and ranks them by audience lost beyond that video's own normal decay.

It also refuses to answer when it can't. Below 100 views a single viewer moves the curve more than a percentage point, so it reports nothing rather than dressing noise up as a finding.

How I built it

TypeScript monorepo. All mathematics lives in a zero-dependency core package; the server only runs binaries and talks to APIs, so the algorithms are testable without a video file or an API key.

Retention curves are treated as a signal: Savitzky-Golay smoothing, central-difference derivative, then PELT changepoint detection on the derivative rather than the raw curve. That inversion is the whole trick — retention is monotone-ish, so segmenting it only finds "it went down again," while segmenting the slope finds changes in the rate of abandonment.

PELT implemented from scratch, O(1) segment costs from prefix sums. No ruptures, no Python sidecar, no ML libraries. Fastify 5, Vite 8, React 19, Tailwind 4, ffmpeg.

Challenges I ran into

The Analytics API returns exactly 100 points per video regardless of length — 7.2s resolution for a 12-minute video, 72s for a two-hour one. Any implementation assuming fixed resolution is silently wrong on long videos.

Severity divides by the median decay rate, which on a sparse curve is exactly zero. It came back as Infinity on real data.

The first version ranked drops by raw audience lost, favouring long segments and reporting four-minute windows that merely contained the actual cliff. Ranking by excess loss over baseline tightened the same detections to 29 seconds.

The hardest problem was restraint, not detection.

Accomplishments that I'm proud of

Both detectors are verified against planted ground truth, not eyeballed. The retention detector recovers synthesised drops at known timestamps 3/3, zero false positives. ffmpeg recovers cuts planted at 10s, 25s and 42s as 10.02, 25.02, 42.02.

Those evals caught real bugs. The first ranking passed 3/3 recall while producing four false positives — nothing in the output looked wrong until it was checked against truth.

Suppressing findings on low-view videos makes the demo less impressive and is the right call anyway.

What I learned

Segmenting the derivative rather than the signal was the insight the project turned on, and it's obvious only in hindsight: a monotone signal has nothing to segment, but its rate of change has everything.

Robust statistics beat clever ones. Three separate bugs traced back to using a mean or a maximum where a median or a percentile belonged.

Ground-truth evals caught what manual checking never would.

What's next for Hindsight

Learned per-creator profiles — training a model per creator on their own back catalogue, with time buckets as training rows. That needs a channel with real accumulated watch time to claim honestly, which is why it isn't claimed here.

Speech rate, since ffmpeg 9 ships a Whisper filter and words-per-minute is a strong correlate of abandonment.

EDL export, so flagged timestamps land on the editor's timeline.

Built With

Share this project:

Updates