BlackBox — Persistent Visual Memory

BlackBox gives cameras persistent memory, letting Alexa+ answer where objects likely are, what changed, and what remains unknown using evidence-grounded visual reasoning.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for BlackBox — Persistent Visual Memory

Inspiration

Most computer-vision systems are good at answering “What is in this frame?” but become much less useful once something disappears from view. I wanted BlackBox to answer a harder question: what should an AI remember after the camera can no longer directly verify something?

A simple example is losing your keys. If a camera saw the keys on a desk, then saw a person interact near a bag, then the keys disappeared and the bag moved, a useful system should remember that history without falsely claiming the keys are definitely inside the bag.

BlackBox is my attempt to build that persistent visual memory layer.

What it does

BlackBox converts video into a persistent, uncertainty-aware representation of the scene. It detects and tracks objects, maintains identities across observations, records temporal state changes, and keeps direct observations separate from inferred information.

A user can ask questions such as:

  • Where are my keys?
  • What changed?
  • What do you actually know?
  • What should I check next?

Instead of returning an overconfident answer, BlackBox provides a ranked hypothesis, the evidence supporting it, alternative explanations, explicit uncertainty, and a recommendation for what to inspect next.

For example, BlackBox may report that a bag is the leading association for missing keys while explicitly stating that it never directly observed the keys entering the bag.

Amazon / Alexa+ Integration

For Build, Ship, Shape, I extended BlackBox with a conversational interface designed around Alexa+ and Model Context Protocol (MCP).

The new integration exposes BlackBox memory through a self-hosted Streamable HTTP MCP server. The server provides tools for:

  • locating a missing object
  • summarizing temporal changes
  • separating observed facts from inferences and unknowns
  • recommending the next re-observation target

An Alexa+-style web simulator demonstrates how a user could naturally ask:

“Alexa, where are my keys?”

and receive an evidence-grounded response from BlackBox's persistent memory.

This conversational layer does not replace the underlying computer-vision system. Alexa+ acts as the interface to a structured memory system that has already tracked what happened over time.

How I built it

The BlackBox backend is written in Python and FastAPI. Video is processed using OpenCV, Ultralytics YOLO, and open-vocabulary vision models. The perception pipeline performs detection, object tracking, persistent identity resolution, temporal event extraction, spatial association, and uncertainty-aware world-state projection.

The query engine reasons over that stored world state instead of asking a model to guess directly from one frame.

The frontend is built with Next.js, React, and TypeScript and displays tracked objects, temporal events, evidence, competing hypotheses, uncertainty, and recommended re-observation targets.

For the Amazon extension, I added a Python MCP SDK v2 server using Streamable HTTP and built an Alexa+-style conversational simulator. Requests map to BlackBox memory tools such as where_are_my_keys_tool, what_changed_tool, what_do_you_know_tool, and recommended_check_tool.

The existing BlackBox system also includes automated tests, restart-safe session persistence, and modular perception/reasoning components.

Challenges

One of the hardest problems was deciding what BlackBox is actually allowed to claim.

An object disappearing does not mean it physically left the scene. Spatial proximity does not prove containment. A tracker ID does not necessarily represent a permanent real-world identity.

Because of this, BlackBox explicitly models the difference between observed, inferred, and unknown information.

Deployment was another challenge. The full open-vocabulary vision pipeline is significantly heavier than the resources available in a small cloud container. For a reliable public demo, the hosted experience replays a preprocessed hero run produced by the complete local BlackBox perception pipeline while preserving its actual world-state and reasoning output.

Accomplishments

I am especially proud that BlackBox is more than an object detector with a chat interface.

It combines:

  • persistent temporal memory
  • object identity tracking
  • structured world-state representation
  • uncertainty-aware reasoning
  • evidence-backed answers
  • re-observation planning
  • a FastAPI backend
  • a polished Next.js interface
  • an MCP-based conversational layer

The system can explicitly answer not only what it thinks happened, but also what it knows, what it inferred, and what it still does not know.

What I learned

Building BlackBox reinforced how different real-world AI is from single-frame perception. Reliable systems need memory, provenance, state management, temporal reasoning, and calibrated uncertainty in addition to strong perception models.

The Amazon extension also showed how MCP can turn a specialized AI system into a reusable conversational capability without collapsing its internal structure into a generic chatbot.

What's next

The next major step is improving BlackBox's visual memory with stronger segmentation and object association, expanding controlled real-world evaluation, and improving the calibration of its uncertainty estimates.

I also want to extend the conversational interface so persistent visual memory can become a reusable capability for smart-home assistants, accessibility systems, robotics, and other physical-world AI agents.

Built With

Share this project:

Updates

Submission history