big-brother

big-brother turns long-form worker POV video into structured, queryable operational memory.

This is a working system prototype, not a polished commercial product. The goal is to show that construction video can become reliable, inspectable memory that supports analysis and decision-making instead of sitting as unsearchable footage.

Why This Matters

Teams record large amounts of field video, but most of it is hard to use after capture. You can watch clips manually, but you cannot quickly answer practical questions like:

What tools were used most over the last 30 minutes?
How much time was spent searching vs executing?
What larger task episodes occurred, and when?

Our thesis: if we convert raw video into a structured timeline of events and episodes, we unlock much better retrieval and downstream automation.

What We Built

Given an input like videos/t1.mp4, the system generates outputs/t1/ with:

windows.jsonl: per-window processing and gate decisions
events.jsonl: atomic structured activity events
episodes.jsonl: higher-level grouped episodes (open/closed/labeled)
summary.json: run-level processing stats
memory.db: SQLite memory used by query APIs and dashboard

The dashboard reads these artifacts directly, so the whole pipeline is inspectable while running.

Technical Approach

We deliberately use a layered architecture instead of one large end-to-end prompt:

Window video (10-20s typical).
Sample representative frames per window.
Compute cheap change signals (motion/drift) for gating.
Extract one structured event per relevant window (Gemini or heuristic fallback).
Build longer episodes from event sequences.
Label closed episodes from sequence context.
Persist to SQLite + JSONL for deterministic querying and auditability.

This gives us lower cost, clearer failure modes, and better debuggability than black-box video summarization.

Core Features (Current)

Multi-video CLI processing with configurable window size, overlap, and frame count.
Gating engine to avoid unnecessary model calls in stable segments.
Structured extraction schema with strict enums (phase, action, tool, etc.).
Rolling state passed between windows for temporal continuity.
Episode boundary detection with heuristic split logic (idle/travel/location/tool shifts).
Episode labeling via Gemini or heuristic labeler.
Deterministic query layer over SQLite (events, episodes, tool-usage, idle-ratio, search-time).
FastAPI dashboard for:
- run snapshot inspection
- frame retrieval at timestamps
- structured queries
- NL ask endpoint powered by SQL agent
- worker/session upload + auto-run kickoff
JSONL streaming output for live traceability.
Heuristic extractor/labeler path for no-key or local fallback mode.

Architecture At A Glance

video -> windowing -> gating -> event extraction -> episode builder -> episode labeler -> storage -> query/dashboard

Main modules:

src/big_brother/video.py - ingest, windowing, frame sampling, change signals
src/big_brother/gating.py - model-call decisioning
src/big_brother/extractor.py - Gemini + heuristic event extraction
src/big_brother/episode.py - episode construction
src/big_brother/labeler.py - episode labeling backends
src/big_brother/storage.py - SQLite persistence (events, episodes)
src/big_brother/query.py / query_cli.py - deterministic queries
src/big_brother/dashboard.py - UI + API surface
src/big_brother/sql_agent.py - read-only LLM-generated SQL answering

For a deeper module-level breakdown, see architecture.md.

Next Iteration

A simple, high-impact next step is connecting memory across runs/videos so an LLM can analyze behavior over longer horizons (cross-day/cross-worker retrieval and orchestration), not just one run at a time.