Flashframe

Upload a locked cut, get a photosensitivity safety certificate in a minute — ClickHouse SQL finds every illegal flash, Gemini names the cause.

Comment

Try it — no signup, no upload, no API key. Open https://flashframe.edycu.dev and click one of the three bundled test clips. The scan runs live against ClickHouse Cloud.

Code: https://github.com/edycutjong/flashframe — all source, the seed-clip generator, the benchmark script, and the SQL. MIT licensed.

Demo video (2:50): https://youtu.be/rPxGyYpVfAE

Inspiration

In 1997, an episode of Pokémon aired in Japan with a four-second red-and-blue strobe sequence. 685 children went to hospital that night. The regulation that followed is why every UK broadcast delivery and every Japanese TV and streaming delivery must clear a photosensitive-epilepsy flash-and-pattern check before transmission — Ofcom Broadcasting Code Rule 2.12, ITU-R BT.1702, and the NAB Japan guidelines.

Here is what that check looks like in practice. A post-production supervisor locks picture on a 90-minute feature at 11pm. Delivery is Monday. The photosensitivity pass costs roughly £300, happens at a specialist lab, and the next slot is Wednesday. So the check happens once, at the very end, when a failure is most expensive to fix — and when it fails, a single flagged strobe run forces a re-edit, a re-render, and a second lab round-trip against a delivery date that has not moved.

That is a measurement problem wearing a compliance costume. A feature is roughly 138,000 frames. The rule is a windowed rule — "more than three flashes in any one second", plus saturated-red runs. Windowed aggregation over a hundred-thousand-row time series is the single thing an OLAP column store is built to do faster than anything else. The check is slow because nobody put it in a database, not because the physics is hard.

Flashframe moves that check from the end of the pipeline to the middle of it, so the lab pass becomes a confirmation rather than a discovery.

What it does

Flashframe screens a locked cut for photosensitive-epilepsy hazards and resolves every violation to an exact frame span.

Zero-config. Three test clips ship with the app. No upload, no configuration, no API key needed to see it work — click a clip and watch the pipeline run.

It finds the hazard and shows its work. ffmpeg pulls per-frame photometrics — full-frame luma plus a 3×3 tiled grid. Those stream into ClickHouse through the official ClickHouse MCP server, and sliding-window SQL pairs opposing luminance transitions into flashes, then merges violating windows into one continuous span. On the strobe test clip it reports frames 740–760 at 6.25 flashes/sec against a published limit of 3.00 — exactly the constructed ground truth the clip generator recorded.

The agent re-samples when a verdict is uncertain. First-pass extraction runs at 10 fps, where the Nyquist limit is 5 Hz — so a 5-alternation flash aliases. On the borderline clip the first pass read 2.08 flashes/sec and looked like a violation. The agent noticed the verdict was borderline and called its resample tool itself, at 30 fps then 60 fps, where the rate resolved to a measured 2.82 flashes/sec with the flashing area under the 25% screen-area threshold. It passes. A single-pass tool reports a false failure on that clip. The same mechanism corrected an understated hazard on the strobe clip, which read 5.0 at 10 fps and resolved to 6.25 after escalation.

Every number says where it came from. The report labels values by provenance: MEASURED (ClickHouse), ADJUDICATED (Gemini), PUBLISHED (Ofcom / ITU-R). Measured values render in monospace, interpretation in proportional type, so you can tell instrument output from model output at a glance.

The safety gate. The flagged clip never autoplays. The control names the hazard before you press it, and reduced brightness is the default. A tool that finds seizure triggers should not fire one at the person reviewing it.

The certificate is written to a ledger table and read back out of ClickHouse, so the record exists outside the app that claims it.

How we built it

ClickHouse — the detection engine. The detection is the SQL. Per-frame luma deltas are evaluated with lagInFrame to pair opposing transitions, so a flash registers as a pair rather than a single brightness change. Windowed accumulations over sliding one-second windows find offending regions, and span merging happens natively so one continuous event reports as one span instead of bucket fragments. Thresholds are joined from a reference table, never inlined as constants — run_chdb_select_query joins the raw ffmpeg CSV on disk against thresholds.csv with no ETL step. All four ClickHouse MCP tools do real work on the demo path: list_databases, list_tables, run_query (read and write), and run_chdb_select_query.

Remove ClickHouse and you need several systems. Replicating windowed transition-pairing over large time series would take a stream processor or hand-rolled windowing code.

Gemini — the adjudicator. The SQL finds candidates, and many are legitimate. When a span flags, the pipeline clips those exact frames with ffmpeg and hands the segment to Gemini with a structured response schema at temperature 0. In a blind test, the SQL flagged a 6 Hz flashing patch and Gemini returned passed=true, naming the cause as a central white square occupying roughly 17.4% of the screen, below the 25% area threshold — the true value is 17.36%. Nothing in the SQL can look at the picture and say what the flashing thing is.

Google ADK drives the loop, using the code-first layer of Google Cloud Agent Builder. The resample escalation is a real tool call the model makes on its own when a verdict is uncertain, not a hand-triggered branch.

The division of labour is the architecture: ClickHouse measures, Gemini judges. The report screen states it, and the certificate's measured value always comes from the SQL.

Stack: Python, FastAPI, Jinja2, ffmpeg signalstats, Docker, deployed on Railway against ClickHouse Cloud.

Challenges we ran into

An off-by-one that inflated every measurement. A span from frame a to frame b lasts b - a + 1 frames, not b - a. The shorter denominator inflated every rate, and proportionally more on shorter spans. After the fix the strobe clip matched its constructed ground truth exactly, and the borderline clip moved from +16.7% to +12.9% error.

A certificate whose headline number changed between runs. The pipeline was writing Gemini's estimate into the certificate's measured field. Two runs of the identical clip produced certificates reading 6.25 and 5.0. A certificate whose number moves is not a certificate, and it contradicted the architecture we were arguing for. The measured value now comes from the detection SQL at the highest fps actually sampled, and Gemini's estimate is stored separately. Two consecutive runs now produce byte-identical certificates.

Gemini could not see the strobe at all. Video sent to the API defaults to 1 fps sampling. A 6.25 Hz strobe is far below Nyquist at 1 fps — it is simply not in the frames the model receives. The same test failed at the default and passed once we passed explicit video metadata at 24 fps.

Aliasing turned a safe clip into a violation. Sampling at 10 fps made a 5-alternation flash read as 2.08 flashes/sec — a false failure. Sampling everything at 60 fps would fix it and cost 6× the extraction and 6× the rows on a 90-minute feature. The resolution was selective escalation: the agent re-samples only the span whose verdict is uncertain.

Free-tier and cold-start realities. The Gemini API tier allows roughly 5 requests per minute, and the resample loop is multi-call, so the pipeline backs off and reports rate limits rather than failing blankly. ClickHouse Cloud auto-suspends when idle and a cold query costs around 25 seconds, so the app issues a warm-up query on startup and tells the user what is happening.

Accomplishments that we're proud of

The measured number matches the ground truth exactly. The generator spliced a 22-frame strobe at a manifest-recorded offset of 57896, and the detection SQL returned 57896 — 10/10 exact detections across two independent runs, zero mismatches. On the strobe clip the measured rate is 6.25 flashes/sec — the number a judge computes from the generator is the number the tool reports. The offset is randomly picked at generation time, so your own run will record a different one and should still match itself exactly.

The agent's escalation is real and it fired on both non-trivial clips — correcting an understated hazard on one and preventing a false failure on the other. That behaviour was predicted in the design notes months before the code existed.

Zero false positives on the clean control clip.

Benchmark on a full feature. 138,240 frames (92 minutes at 25 fps), N=5, ClickHouse Cloud on one replica (8 GiB / 2 vCPU): detection SQL p50 0.55 s, p95 0.94 s on the warm run. That figure is query time only — ffmpeg extraction and Gemini adjudication are excluded from the timed region. We report ingest separately (p50 2.888 s) because it is likely dominated by MCP/HTTP round-trips rather than by ClickHouse, and a single blended number would credit ClickHouse with transport latency it did not spend. The cold run is disclosed alongside it in DEMO.md rather than dropped, including its 14.252 s p95 spike on DETECT, which is a ClickHouse Cloud cold-start artifact.

We disclosed a bias instead of tuning it away. The borderline clip still measures 12.9% high. We did not add a correction constant, because the product's central claim is that thresholds come from published criteria rather than from whatever made the demo look right. A fitted constant would destroy that claim. The bias errs toward over-reporting, which for a screening tool is the safer direction.

The safety gate. The tool that finds seizure triggers refuses to fire one at its operator.

What we learned

Put the measurement in the database and the "hard" problem gets small. A regulation written as "more than three flashes in any one second" is a window function. Once we stopped treating this as a video-processing problem and started treating it as a time-series query, the detection collapsed into one SQL statement.

Sampling rate is a correctness property, not a performance knob. Both of our worst measurement errors — the aliased false failure and Gemini's blindness to the strobe — were Nyquist problems in disguise. Neither looked like a sampling bug from the outside; both looked like the tool being wrong about the video.

Provenance has to be structural, not a label. Our worst defect came from a measured field quietly holding a model's estimate. Separating the fields, and rendering the two differently, is what makes "ClickHouse measures, Gemini judges" a property of the system rather than a sentence in a README.

What's next for Flashframe

  • Per-pixel screen-area measurement, replacing the 3×3 tiled proxy, which is the largest known source of our residual bias.
  • Territory profiles as a first-class control — UK Ofcom, NAB Japan, and the ITU-R baseline differ, and the threshold table already supports the switch.
  • Editor-timeline integration, so the check runs on every cut during the edit rather than on a locked picture.
  • Calibration against a certified lab result, which is the only way to convert a screening-grade tool into a defensible pre-check with a known error bound.

Honest limitations

Flashframe is a screening-grade pre-check against published ITU-R BT.1702 / Ofcom 2.12 / NAB-Japan criteria. It is not a certified lab test and does not claim equivalence to one. It measures luma code value from the decoded signal under an assumed reference display; a certified test measures photometric luminance at a calibrated display.

Screen area is a 3×3 tiled proxy, not per-pixel measurement. Measured accuracy against constructed ground truth is exact on the full-field case and +12.9% on a small-area case, biased toward over-reporting. Gemini's stated figures are visual estimates rather than measurements, and they vary between runs — every one landed on the correct side of its threshold, which is what the product needs from it, but the certificate's measured value never comes from the model.

The thresholds ship in the repo as an inspectable data file (thresholds.csv), cited to source, so anyone can check the arithmetic against the published rules rather than taking ours on trust.

All footage in the demo is synthetic, generated by generate_seed_clips.sh in the repository. No third-party content appears.

Correction to the demo video. The narration at 2:25, the benchmark card on screen at that moment, and the video's original title quote a detection-SQL p50 of 106 ms. A later benchmark on the current code did not reproduce that figure — the measured steady-state p50 is 0.55 s, as stated above and in DEMO.md. The video has not been re-cut, so the older number is still audible and visible in that one scene. Nothing else in the video is affected: the detections, the frame spans, the agent's re-sampling and the certificate are all as shown.

Verify any of this yourself:

git clone https://github.com/edycutjong/flashframe.git
cd flashframe
uv pip install -e . -r requirements.txt
./generate_seed_clips.sh   # regenerate the three seed clips from source
python3 generator.py       # regenerate the 138,240-frame benchmark clip
python3 bench.py           # re-run the benchmark

Live app: https://flashframe.edycu.dev Judge page: https://flashframe.edycu.dev/judge Repository: https://github.com/edycutjong/flashframe Demo video: https://youtu.be/rPxGyYpVfAE

Built With

Share this project:

Updates

Submission history