Scene Dive

Find the moment, not just the movie. Describe what you saw or what someone said, and Scene Dive jumps straight to that exact moment. Built for Fire TV.

Comment

Inspiration

I was browsing YouTube when I came across Google's video about EmbeddingGemma 2, their new multimodal embedding model. I had worked with RAG in school, but only with text. So I thought: what would a RAG-like experience look like for video and streaming? Not "find me a movie", but "find me that moment": the scene you remember but can't find.

What it does

Scene Dive is a TV app where you describe a moment in your own words, what you saw or what someone said, and it finds that exact moment and plays it.

  • The Projection Room: type with the remote or pick a suggestion. A small projector throws the best match onto the screen, with its time, where it sits in the film and, for dialogue matches, the line itself. The other matches wait below as a filmstrip.
  • Search by what was said: paraphrase a line ("someone overheard them talking last night") and playback starts right before it is spoken.
  • Play from the moment: OK grows the projection into the film. The reel then lines up the next moments and says why each one is next ("From your search", "Like 02:11", "Something different").
  • Spoiler guard (optional): moments after where you stopped watching stay blurred until you choose to peek.
  • AI-written moment notes: about half the moments carry a short title and description written offline by a small vision model, clearly labelled AI-written.

Amazon already lets Prime Video users jump to a scene by voice with Alexa+. Scene Dive is my take on that for any library: every moment instead of a curated list, remote-first, what you saw and what was said, on an open model you can host yourself.

What it doesn't do (yet): it doesn't know actors or character names, the catalog is three open Blender films, and the backend runs on your own PC.

How I built it

  • Search: every film is cut into 8-second windows, one every 6 seconds. EmbeddingGemma 2 embeds each window as video, a still frame every 2 seconds, and every subtitle line, all in the same space as the sentence you type. A search is basically "which stored vectors are closest to this sentence?". That takes about 50–140 ms on my RTX 4070, and around 250 ms on a CPU.
  • Backend: Python and FastAPI on my Windows PC, behind a small versioned API that was written first, as a contract between the two halves.
  • TV app: React Native for TV, built on Amazon's multi-TV sample app, and tested and recorded on the Android TV emulator (Android 11, the base of Fire OS 8), which the hackathon FAQ allows.
  • The team: just me, directing two AI coding agents. Claude Opus 5.5 did the TV app, reviews and integration; GPT 6.1 Sol did the backend and retrieval. Each worked in its own git worktree, and nothing reached main without a review and passing tests (640 backend tests, strict typing on both sides).

Challenges I ran into

I joined the hackathon late, so this build leaned heavily on AI development. That meant my desktop was on almost all the time, with the emulator, the AI model and two coding agents sharing 16 GB of RAM (it crashed once). And A LOT of reading: the hackathon rules and forums, everything the agents wrote, and research on the model.

On the technical side: finding a Fire TV path without Fire TV hardware or a Mac, Windows build problems in the sample app (all written down with fixes in a 13-entry friction log), and the AI-written captions. A spot check found about a quarter of them wrong, so AI reviewers checked every remaining caption against its frames, and 62 more were removed.

Accomplishments that I'm proud of

The idea, and the creativity in it. Searching inside films by meaning, by what you saw or what was said, and jumping straight to it felt like something that should exist. Turning it into a little projection room instead of another grid of tiles was the fun part. And it works on real films: on a small test set it finds the right moment in the top 3 for 37 of 37 described scenes and 22 of 23 paraphrased lines (small, and written by an AI agent, but real).

What I learned

AI coding agents like Opus 5.5 and GPT 6.1 Sol are crazy good. They still miss things though, and they like to suggest over-engineered solutions, so I had to push back and keep it simple. I also learned where I help the most: research and judgement calls, like reading through the hackathon's Devpost forums. And the other way around, the agents helped me limit my own scope creep.

What's next for Scene Dive

Popular moments, YouTube-style. With real viewers, Scene Dive could show what people search for: trending searches across the catalog, a "most searched" heat strip on each film's timeline (like YouTube's "Most replayed"), and a "Popular in this film" row in the room. Only aggregated, opt-in counts, never one person's history, and the spoiler guard keeps popular moments after your watch mark hidden.

After that: more films and TV series, voice input, and knowing actors and characters.

Films: Tears of Steel, Big Buck Bunny and Sintel © Blender Foundation, CC BY 3.0. This app was not created or endorsed by Amazon.

Built With

Share this project:

Updates

Submission history