Flipbook

On Fire TV, one press jumps to before a recorded match's next score change, using trickplay thumbnails a Vega player has. No video decoded.

Comment

Inspiration

A fan saves last night's lower-division or women's match for lunch. Nothing marks where the goals are, so they hold fast-forward, overshoot, rewind — and the seek-bar thumbnail shows the score graphic on every scrub, spoiling the result while they hunt for it. Products that skip to goals rely on the rights holder's event feed; Flipbook needs none. A Vega player that shows scrub previews, like Amazon's video sample, has already downloaded something useful: the BIF trickplay file, one small JPEG about every ten seconds of the whole match.

What it does

Flipbook adds a spoiler-aware skip-to-next-score-change key to a long match on Fire TV (Vega OS) — and, if the viewer chooses, ticks on the seek bar — with no feed and no schedule. The scan never decodes video (the player still plays it).

  1. Calibrate once — draw a box around the broadcaster's score graphic with the D-pad. It is remembered per broadcaster.
  2. Scan — Flipbook reads the BIF's thumbnails on the device (JPEG decode in JavaScript), watches that rectangle, and marks every moment it changes and holds. A replay or a brief graphic dropout does not count (shown in synthetic tests; our development match has no replay). Pass 1 reads a sparse subset of the frames (14.9 % on our development match) to put the first ticks up fast; pass 2 still reads every frame.
  3. Jump — one press of ⏩ lands just before the next change: 30 s before the last frame that still shows the old score, so the viewer arrives before the play.

On our development match it found 2 of 2 score changes with 0 false markers and both jumps landed before the play (offline benchmark, labelled by the AI agent that wrote the detector; details under Accomplishments). On held-out match B it landed 2 of 3 goal jumps before the play (AI-labelled truth, n = 3; details under Accomplishments). No Fire TV hardware was available: everything runs on the Vega Virtual Device.

Who it is for. Any viewer who arrives late to a match nobody chaptered, at the cost of one calibration box instead of a feed. It sits in two of the Fire TV priority categories, sports and computer vision (classical image analysis on the thumbnails, no ML). Apps that already show BIF scrub previews, as Amazon's sample does, already hold the data. For a provider, the same scan could run once where the BIF is made so viewers never scan (not built yet; it is on our roadmap in the README), with on-device scanning as the fallback for apps that cannot change their backend.

It is blind by default: the seek bar shows only how far into the match the scan has read, and the score rectangle is masked on every scrub thumbnail. Highlights (the viewer's choice) adds red ticks on the seek bar. Blind mode still discloses three things, which we state rather than hide: a jump tells you a change is ahead, "No later change" tells you none is, and a goal the detector missed is skipped without any sign.

Potential impact

  • Who: anyone who watches a recorded match later — the lower-division, women's, amateur and school matches that usually come with no chapters and no event feed.
  • Why it can ship: it needs nothing a Vega video app does not already have — the BIF file its seek bar already downloads, three KUI SeekBar props, and one calibration box per broadcaster. No event-feed licence, no backend, no ML model.
  • Where it goes: a feature inside any Fire TV sports or replay app that shows BIF scrub previews. For a provider, the same scan could run once where the BIF is made, so viewers never scan (on the roadmap, not built).
  • What is not proven yet: held-out accuracy waits for the human labels on match B; every on-device number is from the Vega Virtual Device; all three matches come from one producer.

How we built it

Flipbook starts from a copy of Amazon's vega-video-sample (MIT-0; taken at upstream commit ef8e4da, upstream history not kept) in React Native / TypeScript. About 4,950 lines of our own TypeScript sit in src/flipbook, test/flipbook and scripts (plus a 119-line HTML ground-truth labelling page in tools/), on top of the sample.

  • Detector (src/flipbook/core/): plain arithmetic on one rectangle — a 64×5 luma grid, a jitter mask, top-3 cell distance and a 3-frame median hold. No ML, no LLM. On our development match it fires 0 false markers (on the match its thresholds were tuned on) where a plain frame differ at the highest threshold in our 8-to-80 sweep that still finds both changes (30) fires 237; a hold-less, mask-less variant of ours fires 348 false markers. Its first version, a 16×4 mean signature, could not separate a goal from noise on that match, so it was replaced.
  • Our own BIF reader. Verifying against Amazon's hosted tv-webinar-hd.bif we found its timestamp-multiplier field holds 0xFFFFD8F0, so the sample's parser would compute about 4.29 billion ms per frame and only works because its lookup ignores the parsed times. Flipbook sanity-checks the header and falls back to the player's video duration, else 10 s; it also rejects a file with no terminator.
  • Vega surfaces in the judged path: the W3C VideoPlayer (playback; the jump sets currentTime); the KUI SeekBar (markers for ticks, displayAboveThumb for the thumbnail mask, partialDisablingConfiguration so Flipbook owns ⏩/⏪); useTVEventHandler and TVFocusGuideView (⏩/⏪ keys, calibration, focus); and AsyncStorage (the score box, remembered per broadcaster). Tooling, not runtime: the Vega Virtual Device and CLI for build, run, logs and recordings. Amazon's sample BIF service downloads the file; one added call hands Flipbook the same bytes.
  • Kiro wrote the BIF writer (scripts/make-bif.ts, src/flipbook/core/bifWriter.ts) and its 18 tests from our brief; we reviewed and fixed its output (commits 024142d, 154bda2; what Kiro did and did not do is in docs/KIRO.md).
  • Pre-registered evaluation: fixtures/PREREG.json freezes the detector parameters, the calibration boxes and the headline rule before any ground-truth file was committed; bench and visibility refuse to run on the two test matches without a committed ground-truth file that names its labeller. The maintainer did not label in time, so the labeller became two isolated AI sessions given only the video; that change is recorded in PREREG.json before any detector run.

Challenges we ran into

  • The Vega packager breaks on a project path containing a directory named assets (we hit it under …/assets/probe/): the vpkg embeds host paths and the app hangs in "Creating surfaces" with no error pointing at the cause (about 90 minutes).
  • The native player refuses http:// URLs and reports only a generic streaming error, so the development match is served as HLS through Shaka. A large MP4 from a redirecting host froze the JS thread and was killed as an ANR.
  • Seeing a goal in a 320-pixel thumbnail is harder than it sounds — hence detector v2 and the fair-baseline ablation: a plain frame differ finds both changes only at a threshold of 30 or lower (of the 8–80 we swept), and at 30, the best of those, it still fires 237 false markers.
  • Honest evaluation. The agent that wrote the detector must not label the data. The test matches were labelled by separate AI sessions that were instructed to open only the video (not the code, the fixtures or any detector output), and the labels were committed before the detector first ran on them. They are AI labels, not a person's, and we say so wherever the result appears.
  • The Vega Virtual Device stopped drawing and then would not boot after the host Mac's screen locked (friction-log entry 10; we think the lock is the cause but have not proven it); with the screen unlocked it booted again, and the calibrated scan of the development match, 19 relaunch runs and the box restored after a relaunch were then captured on it (evidence/vvd/). It also stops drawing text after about 20 minutes of uptime (entry 18), so it needs a restart between long sessions.

Accomplishments that we're proud of

  • It runs on the Vega Virtual Device. On the development match (detector v2, box drawn with the D-pad), pass 1 read 77 of 517 frames, p50 2,771 ms over 19 relaunch runs (p95 3,422 ms), found the same two markers as the offline benchmark, and the box was restored from AsyncStorage after each relaunch. Earlier (2026-10-12), Amazon's own BIF was scanned on the device by the first detector version (v1, before the v2 signature; on that webinar title pass 1 read 203 of 428 frames in 4,085 ms, about 20 ms per decoded frame, with 0 video bytes decoded; that run used a box over most of a webinar frame with no score graphic, so its 34 pass-1 markers are not detections), and a ⏩ jump that landed on the target time without a stacked OS seek (the jump commanded 199.1 s and the player's currentTime still read 199.1 s 1.5 s later).
  • On the development match, in the offline Node benchmark on its BIF (labelled by the AI agent that wrote the detector, so not ground truth; n = 2 changes, 95 % CI for 2 of 2 is [0.16, 1]): 2 of 2 changes found, 0 false markers, both jumps landed before the play (28 s and 30 s), pass 1 read 14.9 % of the frames, and the score box stood out of the noise by 2.64× at 320 px and 2.04× at Amazon's 240×180 letterboxed size.
  • On held-out match B, 2 of 3 goal jumps landed before the play (95 % CI [0.094, 0.992]), with 0 false markers and no jump that showed the new score; the third goal's graphic change scored below the threshold and was missed. The truth for matches A and B was labelled by isolated AI sessions working only from the video, not by a person, and committed before any detector run; parameters, boxes and the 20 s bench lead-in were frozen first (fixtures/PREREG.json; the app's default lead-in is 30 s). On match A, where the lead-in was tuned, 4 of 6 goal jumps landed before the play with 1 false marker.
  • 448 Jest tests pass and 4 upstream tests are skipped (97 of the passing ones are Flipbook's), tsc is clean, and CI, gitleaks scanning and CodeQL analysis run on every push to main and every pull request.
  • An 19-entry friction log (task, steps, expected, actual, severity, workaround, suggestion for each) and five feature requests, written during the build.
  • A landing page that replays the app's own status line from a recorded scan, and a 10-slide deck, both with every number traceable to build/evidence/.

What we learned

BIF files are a contact sheet that Vega apps already hold; most apps use them for one thing. A mask over a thumbnail is a UX decision as much as a pixel one — a blind mode that still leaks "a change is ahead" has to say so. And an honest evaluation plan (frozen parameters, an isolated labeller (a separate Claude session, not a person), a guard that refuses to run early) costs a day and is the only thing that lets us say "held-out" at all.

What's next

A person verifying the AI labels on matches A and B frame by frame, and more held-out matches from other producers (n = 3 goals is small); upstream the BIF parser fixes to vega-video-sample; a packaging-time CLI so a provider can run the scan once where the BIF is made and viewers never scan. No external users yet: the repo is public, and we posted the demo on X and dev.to on 2026-10-13.

Built With

Share this project:

Updates

Submission history