Inspiration
Most computer-vision systems are designed to understand what is visible in the current frame. They are much weaker when an object was seen earlier, disappears, and the system still needs to reason about what may have happened without inventing facts.
BlackBox was inspired by that gap.
The goal is to build a persistent visual-memory layer for cameras and AI agents: a system that does not just detect objects, but remembers what was observed, tracks how the scene changes over time, preserves uncertainty, and distinguishes:
$$ \text{Observed} \neq \text{Inferred} \neq \text{Unknown} $$
Consider a simple example:
$$ \text{Keys visible} \rightarrow \text{Keys become unobserved} \rightarrow \text{Bag moves} \rightarrow \text{Where are the keys?} $$
A conventional system may lose track of the keys entirely or make an unsupported guess about where they went.
BlackBox instead preserves the last direct observation, records the relevant scene changes, ranks plausible hypotheses, and explicitly keeps unknown information separate from inferred information.
For the Nebius × NVIDIA Global AI Hackathon, I extended this idea further:
$$ \text{Observe} \rightarrow \text{Remember} \rightarrow \text{Reason} \rightarrow \text{Identify uncertainty} \rightarrow \text{Choose what to inspect next} $$
NVIDIA Nemotron 3.5 Lightning, served through Nebius Token Factory, acts as an evidence-constrained investigation layer. It reasons over BlackBox's structured visual memory and recommends the next inspection that could reduce uncertainty without turning weak associations into unsupported certainty.
What it does
BlackBox converts video into persistent object identities, temporal events, structured world state, and uncertainty-aware hypotheses.
The core pipeline is:
$$ \text{Video} \rightarrow \text{Detection} \rightarrow \text{Tracking} \rightarrow \text{Persistent Identity} \rightarrow \text{Events} \rightarrow \text{World State} \rightarrow \text{Reasoning} $$
Instead of treating each frame independently, BlackBox maintains memory over time.
For a target such as My Keys, the system can preserve the last direct observation, relevant temporal events, nearby object associations, competing location hypotheses, unresolved unknowns, and recommended next checks.
If the keys disappear while a bag is nearby, BlackBox may rank the bag as the leading association.
But it does not silently convert:
$$ \text{Nearby bag} \Rightarrow \text{Keys are inside the bag} $$
because:
$$ \text{Association} \neq \text{Containment} $$
For the Nebius hackathon, BlackBox also asks:
What evidence should I gather next?
Nemotron receives the structured BlackBox memory and produces a grounded investigation plan that recommends the next useful re-observation.
How I built it
BlackBox is built as a modular visual-memory system rather than a single end-to-end black box.
The perception layer uses OpenCV, YOLO, and open-vocabulary detection to identify relevant objects in video.
A tracking and identity layer then attempts to preserve object continuity across frames:
$$ \text{Detection} \rightarrow \text{Track} \rightarrow \text{Persistent Object Identity} $$
BlackBox then extracts temporal events such as appearance, movement, disappearance, reappearance, and interaction. These events feed into an uncertainty-aware world state.
The backend is built with Python and FastAPI, while the frontend uses Next.js, React, and TypeScript.
For the Nebius integration, I added a server-side inference route that sends structured BlackBox evidence to NVIDIA Nemotron 3.5 Lightning through Nebius Token Factory.
The model is explicitly constrained to:
- avoid inventing observations,
- avoid turning association into containment,
- avoid treating heuristic weights as calibrated probabilities,
- and keep observations, inference, and unknowns separate.
The architecture becomes:
$$ \text{BlackBox Memory} \rightarrow \text{Nebius Token Factory} \rightarrow \text{Nemotron} \rightarrow \text{Investigation Plan} $$
BlackBox remains the source of truth for the visual world state. Nemotron reasons over that state rather than replacing it.
Challenges
The hardest problem was not object detection. It was preventing increasingly powerful reasoning layers from becoming more confident than the evidence justified.
Consider:
$$ \text{Bag near keys} + \text{Keys disappear} + \text{Bag moves} $$
It is tempting to conclude:
$$ \text{The bag contains the keys} $$
but the actual evidence only supports something weaker:
$$ \text{The bag is a plausible associated hypothesis} $$
That distinction shaped much of the architecture.
Another challenge was identity continuity. A tracker ID is not automatically equivalent to a persistent physical object:
$$ \text{Track ID} \neq \text{Physical Identity} $$
Similarly:
$$ \text{Disappearance from view} \neq \text{Physical departure} $$
For the Nebius integration, I also encountered a response-format issue where the initial output budget could be exhausted during reasoning before a usable final answer appeared. I increased the output budget and made the Token Factory response parsing more robust.
Accomplishments that I'm proud of
The biggest accomplishment is that BlackBox behaves like a memory system, not merely a frame-level vision demo.
It preserves the distinction:
$$ \text{What was seen} \neq \text{What is believed} $$
and:
$$ \text{What is believed} \neq \text{What is known} $$
The Nebius integration also goes beyond simply attaching an LLM to vision output.
Instead, it creates an agentic uncertainty-reduction loop:
$$ \text{Observe} \rightarrow \text{Remember} \rightarrow \text{Reason} \rightarrow \text{Find uncertainty} \rightarrow \text{Choose evidence} \rightarrow \text{Re-observe} $$
Nemotron acts as an investigation planner, while BlackBox remains responsible for grounded world state.
What I learned
One of the biggest lessons was that stronger reasoning does not automatically produce more trustworthy systems.
When an AI system operates over incomplete visual evidence, good reasoning sometimes means refusing to collapse uncertainty too early.
A useful visual-memory system needs more than detection accuracy. It also needs temporal memory, identity consistency, provenance, uncertainty, competing hypotheses, and a strategy for gathering additional evidence.
I also learned that large language models become much more useful for physical-world reasoning when they receive structured, evidence-constrained state rather than raw ambiguous context.
What's next
The long-term goal is to move from passive visual memory toward active uncertainty reduction.
Today, BlackBox can recommend what should be inspected next.
The next stage is:
$$ \text{Recommendation} \rightarrow \text{Autonomous Re-observation} $$
That could connect BlackBox to robot cameras, smart-home cameras, wearable devices, drones, or embodied AI agents so the system can actively seek the evidence needed to resolve uncertainty.
The broader vision is simple:
$$ \text{AI that knows when it needs to look again} $$
Log in or sign up for Devpost to join the conversation.