Recall

Ask the stream anything. Recall builds a local multimodal memory of video, speech, and events, then answers viewer questions with timestamped evidence.

Comment

Inspiration

You join a stream halfway through. Everyone understands the plan, recognizes the characters, and knows what happened ten minutes ago. You don’t.

I wanted viewers to be able to ask the stream itself: “What did I miss—and can you show me?”

My primary visual inspiration was Bradley Tangonan’s visualization of video as a three-dimensional object, with two spatial dimensions and time as the third. It made video feel like something you could explore. The Tube project also influenced the interaction: ask a question, highlight matching moments, and inspect the footage.

Recall brings those ideas into a Twitch experience: a stream memory that viewers can question and explore while the broadcast continues.

What it does

Recall builds searchable memory from speech and sampled video frames. In connected Twitch chat, viewers can ask:

  • !recall what's happening?
  • !recall what did they say about the next maneuver?
  • !recall what changed after the first attempt?
  • !recall summarize the stream so far

The bot replies in natural language with a source-time reference. The shared broadcast visualization highlights supporting moments, while the creator can inspect transcripts, frames, and footage in Studio.

The 3D time cube grows as memory accumulates. Viewers can use !cube next or !cube left to navigate the shared visualization. A synchronized timeline and separate evidence player make it possible to revisit a moment without rewinding the broadcast.

A local Twitch-style demo provides /recall questions, clickable timestamp links, and a clip editor for trimming, downloading, and sharing excerpts. These are local clips; Recall does not automatically publish Twitch clips.

The same memory supports additional workspaces: moderators can review surrounding context, campaign teams can collect evidence for deliverables, and editors can shortlist moments. Decisions remain with the person reviewing the footage.

The current demonstration uses a recording replayed through OBS, processed progressively as it plays. It starts with empty memory and cannot answer from footage that has not yet played and finished processing.

How I built it

I developed Recall from a Muse-generated Python baseline, using OpenAI Codex throughout architecture, implementation, debugging, interface development, and testing.

The system has four connected parts:

  • Listening and watching: faster-whisper transcribes audio windows on a separate worker. A local Qwen 3.5 vision model, served through Ollama, describes sampled frames. CPU speech processing can run alongside GPU vision processing.
  • Remembering: timestamped transcripts and observations remain searchable across the processed stream. SQLite text search, an optional local semantic index, and source-linked story notes help recover earlier explanations and plans.
  • Answering: the model interprets the question and receives selected relevant evidence. A separate model pass checks proposed claims, while code validates source IDs and time boundaries. Creator-entered titles, descriptions, and names provide context without pretending to recognize people automatically.
  • Showing and connecting: a TypeScript, Vite, and Three.js interface presents the cube, timeline, chat demo, and review tools. Twitch EventSub receives chat commands, and the Helix chat API sends replies. An OBS Browser Source supplies the replay and its playback clock.

Inference and memory storage stay on the creator’s computer. Twitch carries the broadcast and public chat interaction. No model training or fine-tuning is required.

Challenges I ran into

Keeping memory synchronized. Playback time, completed transcription, and completed visual analysis can advance at different speeds. Recall needs to distinguish what has played from what it has actually understood. The same time boundary now applies to answers, citations, cube contents, and replay ranges.

Making local processing practical. Early runs slowed sharply and accumulated a backlog. I added separate speech and vision workers, bounded queues, lighter model settings, and diagnostics for processing time, memory, GPU clocks, and power. Skipped or failed analysis is recorded as missing coverage.

Remembering the whole story. Recent frames alone produced generic answers. Retrieving earlier speech, retaining source-linked story notes, and supplying creator context helped connect the current scene to previous explanations.

Recovering precise spoken facts. A question could match one transcript fragment while the actual time or number appeared in the next. Retrieval now includes eligible neighboring fragments in chronological order. Spoken facts also have a separate answer contract from uncertain visual interpretations.

Handling uncertainty usefully. Sampled frames can miss brief actions, and transcription can mishear speech. Recall distinguishes spoken statements, visual observations, and cautious inferences. Its verification step reduces errors, but does not guarantee correctness.

Accomplishments that I'm proud of

Recall connects local multimodal processing, searchable history, a growing visualization, and real Twitch chat in one working prototype.

The current build passes 211 automated tests, covering retrieval, time boundaries, processing recovery, Twitch transport, and other application behavior. Targeted checks recovered specific spoken details—including a distance of 152 kilometers and a maneuver time of 6:50 a.m. Pacific—while an unsupported question about landing on Mars returned uncertainty.

I’m especially proud that the answer remains connected to its source. A viewer can ask, understand, and inspect.

What I learned

A stream’s memory needs more than a large prompt. It needs organized history, useful retrieval, clear timing, and reliable recovery when processing falls behind.

I also learned to separate what the creator tells the system, what the models observe, and what the recording actually establishes. That distinction makes answers more useful without making them sound more certain than the evidence allows.

What's next for Recall

Next, I want to support direct live-source ingestion, evaluate accuracy across more stream categories, and improve processing efficiency without losing brief but meaningful events.

I also want to develop individual viewer exploration through a Twitch Extension, stronger speaker attribution, and richer moderator, editor, and sponsorship-review workflows.

The goal is simple: make a stream’s past accessible while its present keeps moving.

Credits

  • Bradley Tangonan: primary visual inspiration for exploring video through space and time.
  • Tube: inspiration for connecting questions, visual search, and source evidence.
  • Louen Pottier’s LaGSplat: an additional research and visual reference.
  • Muse: the initial Python baseline that Recall extended.
  • OpenAI Codex: development assistance throughout the build.
  • NASA: spacecraft footage used during development.

The referenced visualization projects inspired Recall’s design; their source code and assets were not copied into the application.

Built With

Share this project:

Updates

Submission history