FrameSheet

Inspiration

Inspiration

Warehouses, retail counters and security feeds generate hours of video that never get audited. Ask "how long did that customer actually wait?" and the only honest tool is a human scrubbing a progress bar by hand.

Meanwhile, operations teams live in spreadsheets. A cell can compute anything — but it can't prove anything. A video can prove anything — but can't be queried.

FrameSheet stitches these two incompatible organs into one Frankenstein: a fully client-side computer-vision watchdog that detects zone activity and feeds a reactive spreadsheet in which every number is clickable, replayable video evidence.

What it does

Draw zones on any video (or one of 4 built-in feeds). The engine detects activity inside each zone — motion frame-diff mode works on ANY footage, object mode tracks person/vehicle/package blobs on-device — and carves it into timestamped events: start, end, duration, confidence.

A live formula grid aggregates them:

=COUNT_EVENTS("Zone_A")  =AVG_DURATION("")  =MAX_DURATION("")
=COUNT_OVER("", 20)      =IF(B4>25,"BOTTLENECK ALERT","NORMAL")

Click any event row, any timeline bar, or a proof-linked formula cell — the video jumps to that exact moment, flashes the zone, and replays the segment. Click-to-Proof.

Built for judges to verify in 30 seconds: 3 zero-download synthesized CCTV scenarios, a bundled real 38KB card-terminal clip (pre-tuned), drag-drop your own MP4 (never leaves the browser), one-click presets, area-aware auto-calibration, CSV/Sheets export, and a CV-mask overlay that shows exactly which pixels the engine reacts to.

How we built it

100% client-side TypeScript/React — zero backend, zero media APIs, offline.

Detection core is pure and property-tested: an IDLE→ARMING→ACTIVE→COOLING hysteresis segmenter (retroactive start anchoring, gap-merging, overlap dedupe) over a 192×108 luma grid; object mode runs background subtraction → 3×3 dilation → connected-component labeling → aspect/size classification → centroid tracking with background-anchor freezing so stationary dwellers never get absorbed into the background model.

The three CCTV "feeds" are pure render(t) canvas functions — deterministic, seekable, asset-free. The spreadsheet ships a tokenizer, recursive-descent parser and evaluator with cell refs, IF, arithmetic/comparison, cycle detection and autocomplete, plus proof linkage (MAX/MIN/COUNT_OVER return the event ids that produced them).

Whole app = one 443KB single-file build; real footage is a 38KB inlined clip; no paid APIs anywhere.

Challenges we ran into

Two real bugs, both caught by our own verification stack — and both in git history:

1. fast-check property tests caught a cell-reference false-positive in the spreadsheet's cycle detection, blocking legitimate formulas like =IF(B4>25, ...). The shrunk counterexample mapped to a spec invariant.

2. Our demo-recording automation exposed that a standing customer pixel- shrunk to aspect <1.35 and the per-frame classifier flipped them to "vehicle", killing person-gated events. Fixed with sticky track classification — a track keeps its spawn-time class for its lifetime.

Also tuned retail sprite panels so dark trousers contrast the floor and survive the foreground mask.

What we learned

Detached, pure engines + property-based tests beat ad-hoc demos — our segmenter and formula engine were validated on 100+ generated cases before any UI existed. Classical CV (bg models, morphology, centroid tracking) is surprisingly effective inside the browser when you control grid size and anchor-freezing. Deterministic pure render(t) procedural scenes make reproducible proofs — crucial for Click-to-Proof.

Accomplishments that we're proud of

• Turned CV into an auditable time-study table — every cell is a clip. • 15/15 tests ##

Built With

• canvas
• computer
• css
• fast-check
• kiro
• react
• tailwind
• typescript
• vite