How video becomes evidence

Solution architecture: visual and audio evidence run in parallel, fuse into one score, and end at a staff decision that nothing bypasses.
Labels: staff define and validate the vocabulary themselves. Each carries a visual prompt plus counts of waiting, accepted and rejected.
Audit log: Every action on the data is recorded and cannot be edited or deleted. Staff identifiers and one participant name are masked here.

Inspiration

Video intelligence is what drew us in, because it is the harder problem. You cannot hand an hour of unedited, low-light, handheld footage to a model and ask what happened. You have to decide where the moments are before you can say what they contain, and the moments TPA cares about only exist across time: a participant taking the stage for the first time, sustaining focus through a full activity, going quiet before a first verse and then being met with applause. Multi-gigabyte files with no cloud and no GPU budget pushed us toward combining several weak signals, sight, sound, speech, and people counts, which turned out to be the right shape for the problem anyway. What convinced us the work mattered is that the answer is not a better classifier. A staff member still has to watch the clip, since only they know the participant and the session, so the model's job is to cut an hour down to the thirty seconds worth watching. The thing worth keeping is the decision that staff member makes, recorded next to the evidence that produced it: the grant evidence TPA needs now, and the training data nobody has yet.

What it does

TPA's "This Is ME" programme records hours of youth arts footage: workshops, rehearsals and final shows. Staff need evidence of participation and engagement for funders, but nobody has time to watch and code every minute by hand.

Our solution is a private, staff-in-the-loop video review workspace that runs entirely on one computer. It watches and listens to every video, splits it into moments, and suggests what is happening in each using TPA's own activity codebook: music, dance, theatre, conversation, instruction, fashion, visual arts, writing, and more. It also suggests participation, context, and observable indicators.

Staff stays in charge. Every suggestion lands in a Review inbox with the evidence behind it: what the camera saw, what the microphone heard, the transcript, and how many people were on screen. A reviewer accepts, rejects, revises, defers, or adds a label with one key. Only staff-validated labels count in summaries and exports. Staff can also:

search moments across the whole archive
ask a question about a video and get an answer with clickable timestamps
follow model quality on a feedback-loop page (precision, recall, F1)
export pseudonymous training data
see an append-only audit log of every action taken on the data

How we built it

Ingest. Videos are imported from a folder or uploaded with a recorded permission scope and a SHA-256 hash, so every result traces back to an exact file.

See and hear. Background workers detect and track people (RF-DETR and ByteTrack, two samples per second) and estimate pose (RTMPose). They classify sounds (AST), transcribe speech (Whisper), and embed frames (SigLIP). Scene cuts and audio changes split each video into reviewable moments.

Suggest. Each label's score is a weighted blend of vision (SigLIP similarity to a plain-language description of the label), audio (AST sound groups), and simple people rules. A local language model (Qwen via Ollama) then re-reads all the evidence and the transcript and refines the ranking. It marks primary and secondary activities, because real moments have several things going on at once.

Identity. Tracks stay inside a single video. ByteTrack links a person's detections across frames so staff can see time on screen for one track. Nothing links a person across videos. There is no facial recognition, no face embedding, no identity registry, and no biometric matching of any kind.

Review and trust. A Next.js interface with keyboard-first review, timeline, search, and "Ask this video". A FastAPI service records every change in an append-only edit log and every action in an append-only audit log.

Measure. We turned TPA's own spreadsheet annotations for 9 videos into an answer key and scored the model against it. That is separate from the live feedback-loop metrics built from reviewer decisions.

Privacy by design.

separate partner and public-demo instances, and role-based accounts
no internet calls at any point, and no age, gender, or race inference
no facial recognition, face embeddings or cross-video identity linking
an append-only audit log so every action on the data is accountable after the fact

Challenges we ran into

Privacy versus building. The footage shows real young people, so we set a hard rule: no developer or AI assistant views partner footage. We built and tested on public sample videos and disposable database copies, and evaluated partner results only through aggregate numbers.

Long, messy recordings. Real files run up to about 12 GB and over an hour, with rotation and odd timestamps. We added folder import, rotation correction and a job queue in the database that survives restarts.

Activities overlap. A moment is rarely just one thing: conversation happens during visual arts, music during theatre. We moved from one label per moment to primary and secondary labels, and reviewers can add several.

Confidence is not accuracy. Model scores looked high, between 90 and 100 percent, while agreement with staff was much lower. We now report the model's self-confidence and its measured accuracy separately, and say so plainly in the interface.

Where we stopped. We designed a permission-gated path for cross-video identity before building it, then cut the feature entirely. The rules for this data prohibit facial recognition and identity registries, and we judged that a permission record does not make that line safe to approach with youth footage.

CPU-only compute. Our first pass decoded every frame. Seeking straight to the frames actually needed cut that work drastically.

Accomplishments that we're proud of

Processed all 12 partner videos (3.85 hours, 26 GB) end to end on a laptop in about 4 hours 16 minutes, with zero failed jobs and no data leaving the machine.
8,889 label suggestions across 1,338 moments, each with explainable evidence, plus 60,827 people detections (average confidence 0.87), 6,940 audio windows and 143 minutes of transcribed speech.
An honest baseline against TPA's own annotations (467 moments; MUSIC F1 0.51, instruction 0.44, fashion 0.40) instead of an unverifiable accuracy claim.
Human-in-the-loop by construction: nothing counts until staff validate it, and every action is audited.
A design that works entirely from activity, sound, speech and people counts, with no biometric identification of any participant.

What we learned

In youth programmes, trust and consent matter as much as model quality. Privacy constraints shaped the architecture from day one, and the most important engineering decision we made was to remove a feature.
Several weak signals beat any single model: sight, sound, speech and people counts together. A local LLM is good at weighing them when it is told to cite its evidence.
Self-confidence and accuracy are different things. Measuring against staff annotations kept us honest about where the model struggles, which is quiet activities like writing and visual arts.
Reviewer speed is a feature. Keyboard shortcuts, evidence chips and most-confident-first ordering are what make a large queue manageable.

What's next for How video becomes evidence

Learn from staff decisions. Use the accepted, rejected and added labels to recalibrate scores and fine-tune small per-label classifiers, then re-measure against TPA's annotations.
Use programme context. Import TPA's session schedules and timetables and tag each file with its session type (workshop, rehearsal, final show), so suggestions know what is plausible.
Two-staff pilot round. Two staff code the same sample, to measure how far reviewers agree with each other (Cohen's kappa) and to set realistic targets.
Calibrated confidence. A random-sample review mode that turns scores into real probabilities and estimates precision without reviewing everything.
Deploy with TPA. A packaged on-site install with retention schedules and an annual permission renewal.