Inspiration
Most accessibility tools focus on describing what is happening visually right now. But media is not experienced one moment at a time. A blind viewer may understand a scene when it is described, yet still lose track of where an object was placed, who entered earlier, what changed between scenes, or where something eventually ended up.
We wanted to explore a different question:
What if a blind viewer could ask a movie what it remembers?
That idea became RECALL, a persistent visual memory agent for blind and visually impaired people.
What RECALL Does
RECALL watches a video and converts important visual events into structured, timestamped memories.
Instead of storing one large summary, it remembers individual events such as:
- A character entering or leaving
- An object being placed somewhere
- An object being moved
- A character performing an action
- A location changing
- Something appearing or disappearing
- Important changes between moments
These memories are stored in ClickHouse as a temporal event history.
A user can then ask questions naturally:
Where did the envelope go?
Who moved it?
What changed after Maya left?
Was the envelope still on the table later?
RECALL retrieves the relevant events through the ClickHouse MCP server and uses Gemini to reason over their temporal relationships before generating an answer.
How We Built It
The system is built around a simple agentic pipeline:
Video → Gemini → Visual Events → ClickHouse → ClickHouse MCP → Gemini Agent → Answer → Voice
Gemini's multimodal capabilities are used to understand the video and extract structured visual events with timestamps.
ClickHouse acts as RECALL's long-term visual memory. Instead of asking the model to remember an entire video inside a conversation, the important events are persisted as structured data that can be queried over time.
The RECALL agent uses Google ADK and the ClickHouse MCP integration to retrieve evidence from this memory. The agent can formulate focused queries based on the user's question and reason over the chronological results.
The final answer can also be spoken aloud, making the experience accessible without requiring the user to continuously inspect a screen.
Why ClickHouse Matters
ClickHouse is not simply being used as a database behind the application.
It is the agent's queryable temporal memory.
When a user asks a question, RECALL retrieves the relevant events from ClickHouse through MCP. This allows the agent to reason about relationships such as first, last, before, after, still, no longer, and changed.
For example, instead of generating an answer from a generic video summary, RECALL can retrieve:
- Maya placed the envelope on the table at 00:05.
- Daniel picked it up at 00:18.
- Daniel placed it inside a drawer at 00:42.
The agent can then conclude that the envelope ultimately ended up in the drawer.
What Makes RECALL Different
RECALL is not designed to continuously narrate every frame.
It creates a persistent, queryable memory of visual events.
That changes the interaction from:
"Tell me what I can see."
to:
"Remember what happened and let me ask about it."
This temporal memory model can eventually extend beyond movies to theatre, livestreams, sports, concerts, educational videos, and other visual media.
Challenges
The biggest challenge was preventing the system from turning a video into an unreliable generic summary.
We needed the visual information to remain structured, timestamped, and queryable. We therefore designed the extraction layer around discrete events rather than a single narrative description.
Another challenge was making the agent actually use the partner technology. ClickHouse is part of the reasoning loop through MCP, rather than simply being mentioned in the architecture.
We also designed the interface around accessibility, with keyboard navigation, screen-reader-friendly controls, large interaction targets, spoken responses, and timestamp evidence.
What We Learned
We learned that the most useful AI systems do not necessarily need to describe everything.
Sometimes the more powerful capability is remembering the right things and retrieving them at the right moment.
RECALL explores that idea by combining multimodal Gemini understanding, agentic reasoning, MCP-based tool use, and temporal data infrastructure into one accessibility-focused media experience.
What's Next
The current prototype focuses on short-form video and a controlled demonstration environment.
Future versions could support live media, longer videos, personal visual memories, theatre and concert experiences, livestreams, and richer temporal relationships between people, objects, and locations.
Our long-term goal is simple:
Make visual media something people can remember and question, not something they must be able to see.
Log in or sign up for Devpost to join the conversation.