Observe

Inspiration

Researchers often spend hours manually reviewing video footage to identify behaviors, measure events, and record observations. While AI can automate parts of this process, most existing tools focus on generating descriptions or predictions without making it easy to verify exactly where those conclusions came from.

We wanted to build something different: an AI-assisted research tool that doesn't just produce answers, but makes those answers inspectable, correctable, and reproducible.

This led us to create Observe, a human-in-the-loop video analysis platform designed to turn raw footage into structured, evidence-backed research measurements.

What it does

Observe helps researchers transform video footage into reliable, reviewable datasets.

Instead of simply asking AI to describe a video, users can define specific observable behaviors, analyze footage, and inspect the exact video frames supporting each detected event.

Our platform provides:

AI-assisted event detection: Identifies candidate behavioral events and their timestamps using computer vision.
Evidence-linked observations: Connects each measurement directly to its supporting video frames.
Human-in-the-loop review: Researchers can accept, reject, adjust, or manually add observations rather than blindly trusting model predictions.
Uncertainty and coverage tracking: Distinguishes reviewed footage from unreviewed or uncertain segments, preventing missing observations from being mistaken for confirmed absences.
Correction propagation: Updates downstream measurements when researchers correct underlying annotations.
Revision history: Preserves changes so researchers can understand how their dataset evolved.
Reproducible exports: Generates structured event data, feature measurements, and metadata that support independent analysis.

Our initial use case focuses on controlled tabletop interactions, including tracking when an object enters a defined region and when a visible hand approaches an object.

The goal is not to infer someone's intentions or emotions. Instead, Observe focuses on measurable, observable evidence that researchers can independently verify.

How we built it

We designed Observe around a research-oriented video analysis pipeline that combines computer vision, temporal event processing, and human review.

Using Python, we process uploaded videos to extract frame-level information and preserve source timestamps. We use pretrained computer vision techniques, including MediaPipe hand landmark detection, to identify