-
-
Signal Review Reel
Inspiration
The demo video is the one artifact a judge reliably watches, and it is the hardest thing to check the description against. The description lists eight capabilities; the video shows three screens. Nobody has time to scrub a recording against a bullet list, so the gap between what a project claims and what it demonstrates usually goes unexamined.
There was also a concrete hole to fill. A sibling project, Signal Packet API, reads written and repository evidence and explicitly records the demo video as a gap because it cannot watch it. Video is where generative media is actually the right tool: to review a video honestly you have to transcribe it, look at frames, cite timestamps, and then produce something a human can watch in a minute. That is a pipeline, not a prompt.
What it does
A project name, its description (the claims under review) and its demo video go in. Three things come out:
- The review reel, roughly 45 to 60 seconds, with narration, baked-in captions, verdict badges, the source frames shown large, timestamps, and a closing scorecard
- A Signal Packet, machine readable, pairing every claim with its evidence and verdict
- A provenance record, listing every generative step and every artifact with its hash
Five stages produce it. Ingest stores the originals on B2. Analyze pulls audio and evenly sampled keyframes with ffmpeg and transcribes the narration with timestamps. Audit judges each claim from the description strictly against the transcript and frames. Narrate writes a grouped review script and speaks it. Publish writes the packet and the provenance next to the reel on B2.
The verdict vocabulary is four values, and the distinction between two of them is the point: shown, not_shown, contradicted, unclear. not_shown means a concrete claim is simply absent from the demo. unclear is reserved for evidence that is present but genuinely ambiguous. Neither one means false, and the reel never says false.
Evidence integrity is enforced in code, not requested in a prompt. A cited transcript quote has to actually occur in the transcript. A cited frame timestamp has to exist inside the video. When either check fails, the claim is downgraded. The model's citations are verified, never trusted.
How we built it
The web app is a thin shell around the five-stage pipeline. Two rules shaped the whole architecture and both are visible in the diagram.
Every generative step is a Genblaze pipeline, never a raw model call. Four pipelines run through Pipeline(...).step(...) with ObjectStorageSink: transcription, claim audit, review script, and TTS. Each output asset lands on B2 with a SHA-256 and a hash-verified canonical manifest, and provenance.json indexes all four with run ids, canonical hashes, manifest URIs, per-step models and timings. Input assets are passed with pre-computed SHA-256 through external_inputs=, so manifests and step-cache keys stay stable across reruns.
Genblaze needed two custom provider adapters to do this, both built on its SyncProvider lifecycle and tested offline with stub clients. genblaze-google ships video (Veo), image (Imagen) and text-only chat, but there is no way to feed media into Gemini and no audio modality at all. So GeminiMultimodalTextProvider takes audio, image and text in and returns text, and GeminiTTSProvider turns text into spoken WAV narration.
Every artifact of record lives in one private B2 bucket. Originals, keyframes, pipeline manifests, the reel, the captions, the packet, the provenance, and the job state that lets a job survive a restart. Storage access goes through Genblaze's S3StorageBackend.for_backblaze(). The browser only ever sees short-lived presigned URLs, so the bucket stays private and the UI still renders everything.
Around that: FastAPI for the job API, ffmpeg and ffprobe for audio and keyframes, Pillow plus ffmpeg for reel assembly, and a single static HTML page for the frontend with no build step.
Challenges we ran into
Genblaze had no path for what we needed. Media in and audio out were both missing from genblaze-google. Writing two provider adapters was the largest single piece of work, and it turned out to be the most reusable one; there is a draft upstream issue in docs/ for it.
Reel assembly without ffmpeg's drawtext. Depending on drawtext means depending on a specific ffmpeg build. Instead Pillow renders every caption, badge and label to an image and ffmpeg composites them. That also made the visual work easier: aspect preserved, dead letterbox trimmed, blurred fill behind the frame, a restrained slow zoom, quick-cut montages, and TTS segments trimmed and loudness-normalized so the pacing holds.
Two verdict vocabularies that both had to be right. The review needs four values to be fair to a project. The shared Signal Packet schema, vendored identically across all three Signal Packet projects, carries a coarser three-value status so packets stay interchangeable. Rather than pick one, the packet carries both: verdict is the review's actual finding, status is derived from it, and tests/test_packet_consistency.py pins the two together.
Recording the submission video reliably. A live generative pipeline is a bad thing to demo on camera: it takes minutes, costs money, and differs every take. make demo runs the app with DEMO_MODE=1 and replays one real committed run on a compressed timeline, so every take is identical and works offline on a clean clone with no credentials. Nothing on screen is fabricated: the reel, captions, transcript, packet and provenance in demo/out/ are one real job's genuine outputs, and you can prove it yourself by hashing the committed sample video against the input SHA-256 in that run's provenance. Only the waiting is simulated.
Choosing a sample that could embarrass us. The easy demo is a project whose description matches its video. The useful one is not. The Save Pip sample overclaims in ways its own demo cannot prove, and the reel marks those NOT SHOWN.
Accomplishments that we're proud of
Provenance that a reviewer can independently verify. Four pipelines, four manifests, canonical hashes, input SHA-256s, per-step models and timings, rendered as an ordered timeline rather than dumped as JSON.
make setup && make demo works on a clean clone with no keys, no network and no B2 credentials, while make dev runs the real thing end to end. A reviewer can see the product in a minute and still evaluate the actual pipeline.
Two Genblaze provider adapters that fill real gaps in the SDK, tested offline, and written to be upstreamable rather than bolted on.
Three rendered result views: the reel with its captions and transcript, the packet as claim cards with evidence keyframe thumbnails labelled by source timestamp, and the provenance timeline. Raw JSON is a secondary action on each, never the default.
The scorecard counts at the end of the reel are computed from the verified verdicts, so the summary cannot drift from the findings.
What we learned
Provenance is nearly free if you build it into the pipeline and expensive if you add it afterwards. Passing pre-computed input hashes and letting the sink write manifests meant the provenance file was mostly an index over work already done.
Provider adapters are the highest-leverage contribution to Genblaze, which its own contributing guide says outright. Once the two adapters existed, adding a fourth pipeline stage was routine.
"Not shown" and "false" are different claims, and if that distinction is not visible in the reel itself then the tool is unfair to the project it is reviewing. This drove the vocabulary, the badge design and the wording of the narration.
Environment fragility is worth engineering away early. Baking ffmpeg and the caption fonts into the Docker image removed an entire class of "works on my machine" failure.
What's next for Signal Review Reel
Upstreaming both provider adapters to backblaze-labs/genblaze, since the multimodal and TTS gaps affect anyone building media-in pipelines.
Longer videos with chaptering, and per-claim deep links that jump to the exact source timestamp instead of showing a frame.
An API and webhook mode so an evaluation pipeline can submit jobs and collect packets without the browser, and feeding the resulting packets into Signal Evidence Graph so video evidence joins written evidence in one lineage.
Log in or sign up for Devpost to join the conversation.