VISTA: Visual Intelligence and Spatial Temporal Archive

Ask your video where things are. Every answer comes with the frame that proves it.

Comment

Inspiration

I lose things. Phone chargers, keys, a coin that rolled off a desk. Video cameras see all of it, but nobody can ask a video "where did it go?" and trust the answer. I wanted a system that answers with the exact frame that proves it, and that admits when it doesn't know.

What it does

VISTA (Visual Intelligence and Spatial Temporal Archive) turns a video into a persistent, queryable memory of objects. Upload a clip and ask "where is the laptop?" The answer cites a frame, and clicking it jumps the video to that moment. Observed facts are kept separate from inferences, and "I cannot determine" is an allowed answer.

A verify agent then checks each answer. It crops the claimed region, zooms in, and re-runs detection. Depending on what it sees, it accepts the answer, tries another candidate, re-scans at higher sampling, or asks the human. It never guesses a location, and every step is shown as a trace in the UI.

How I built it

  • OpenCV 5: video decode, adaptive frame sampling with optical flow and background subtraction, YOLOv8n detection through cv2.dnn, ORB and histogram re-identification, Kalman tracking, and zoom crops for verification.
  • AWS: an EC2 Graviton5 instance running the Cloud Optimized OpenCV (COOL) image with OpenCV 5.1, Amazon S3 for video, memory and evidence, a bucket-scoped IAM role, a one-IP security group, and a budget alert. All of it is scripted in the repo.
  • Stack: Python, FastAPI, React and SQLite, with dependencies pinned in uv.lock.
  • Agent: a deterministic loop of lookup, crop and re-detect, re-scan, then ask the human. It uses no language model.

What I measured

  • Graviton ran the pipeline about 1.3x faster and 31% cheaper per frame than x86.
  • COOL matched stock OpenCV on the same Arm hardware, within noise (+2.8%). I report that as measured.
  • On 20 real phone videos, the first answer was right 75% of the time on my laptop and 60% on the deployed instance. I haven't isolated why.
  • On four MOT17 sequences, MOTA ranged from 0.15 to 0.53. Precision was high, and recall was low on crowded 1080p scenes.
  • On held-out clips, the agent's automatic accuracy was about equal to the plain answer. It doesn't improve accuracy. It makes answers checkable and handles failure safely.

Challenges

I first blamed one poor result on objects fragmenting into many tracks. Looking at the data showed the real cause was persistent false-positive boxes from a patterned table, so I corrected my own write-up. I also found that my first crop check stretched thin objects like bottles, fixed it, and then tested on clips I hadn't touched. The weakest spots are a small evaluation sample, a closed-set detector that struggles with flat objects, and a printed photo of a face being reported as a person.

What I learned

Measure before claiming. Several of my first guesses were wrong, and the honest numbers were more useful than flattering ones.

Responsible use

VISTA stores video and per-person tracks. Use it only on footage you have consent to process.

What's next

A larger detector for crowded scenes, and isolating the gap between the laptop and the deployed instance.

Share this project:

Updates

Submission history