CAPTIONVERSE
Tagline: Follow the speaker. Catch the sound. Stay in the story.
Inspiration
When several characters speak at once, a line of subtitles at the bottom of the screen can leave a viewer searching for the person behind the words. A sound outside the frame can be just as easy to miss. Captionverse began with a simple wish: let captions help viewers follow the scene as it unfolds, and let them return to the exact moment when they lose a line.
What it does
Captionverse plays captions alongside their source video and can place dialogue near a supported speaker, show familiar bottom subtitles, or present selected moments in manga-style bubbles. The TV viewer lets someone ask who spoke, retrieve an exact quoted line, and jump back to its time in the clip. A separate player offers larger text, intentional reading pauses, interval replay, and a way to correct or review captions. The answers come from recorded caption cues. A clearly labeled simulated Alexa client demonstrates voice-style control by calling the viewer's MCP tools.
How we built it
We built an offline Python pipeline that segments speech, groups speaker turns, drafts transcripts, and proposes sound events. It uses sherpa-onnx and WeSpeaker for speaker grouping, faster-whisper for transcription, and YAMNet for sound candidates. For our Tears of Steel example, face detection and audiovisual speaking scores help place a caption by a visible speaker when the evidence supports it. Node.js serves the clip, caption record, viewer, and control tools. The browser checks that the captions belong to the selected source, lays out timed text, and can render captions as WebGL planes. We also built a Vega WebView shell for the TV experience.
Challenges we ran into
Speech recognition becomes less reliable when voices overlap or music covers a word, so the viewer needs to carry uncertainty through to the caption instead of hiding it. A face visible in a shot may belong to a listener, and film dialogue is often mixed in the center, where stereo position gives little help. The final official Vega Virtual Device demonstration shows moving licensed film pictures with synchronized captions, pause, speaker recall, an exact-line jump, source return and continued viewing after reopening the app. To sustain that picture transport, we used 1,056 verified silent JPEG source frames. The captions and speaker grounding are prepared, and the video narration is editorial. The comparison segment is clearly labelled as an earlier native screenshot because the latest comparison capture had a corrupted pane. Native movie decoding and source audio still need further work.
Accomplishments
We made a playable film example from a licensed Tears of Steel excerpt and connected its dialogue record to speaker placement, caption-style comparisons, exact-line recall, and seeking. The example includes moments where a speaker can be anchored to a face and moments where the speaker remains off screen or uncertain. The ordinary player extends that experience with reading controls and reversible review. We retained the source and processing details needed to understand how the example was made, and we exercised the TV wrapper with remote control on the official Vega Virtual Device.
What we learned
A spatial caption is useful only when its position has a reason. Our stereo example could place isolated voices near their left or right positions, while the film example needed audiovisual evidence to handle centered dialogue. We also learned that a caption record must stay tied to the exact media it describes. Keeping source identity, caption revisions, and playback timing distinct makes it easier to catch a mismatch before a convincing caption appears beside the wrong scene.
What's next
We want to improve overlapping speech, follow speakers through longer shots, and locate meaningful sounds more reliably. We also want to study whether viewers find scene-based captions easier to follow and how much reading effort they add. Future work could support more languages, personal reading preferences, and richer questions grounded in the clip. Further testing will focus on native movie decoding, source audio and physical Fire TV devices.
Log in or sign up for Devpost to join the conversation.