FFL: Heck Flashing Lights

Flashing lights hurt your eyes? Hate when videos don't come with a proper photosensitivy/epilepsy warning? Heck them flashing lights with this hack :3

Comment

Inspiration

Modern video platforms often lack detection of highly flashy sections in their videos, which can be dangerous for people with epilepsy or other flash-induced conditions. By providing warnings or allowing users to skip these segments, we can help create a safer viewing experience for individuals who are sensitive to such stimuli.

What it does

  1. Scans current webpage
  2. Makes a call out to our server with YouTube URL of currently playing video
  3. Server will download and extract frames using ffmpeg
  4. Parses consecutive frames to highlight spots with flashing lights, returns to client as response
  5. Client (browser extension) will then decorate video with warnings of when flashing light sequences occur.

How we built it

  • Core logic (Python)
    • yt-dlp used to download YouTube video to .webm format
    • ffmpeg used to extract image frames from downloaded video
    • OpenCV used to create convolution kernels, which are used to scan images for value (HSV) diffs
  • Server (Python + Ngrok)
    • Manually created Websocket event handler
    • Asynchronous + Multithreaded server handler which allows for multiple simultaneous connections
  • Browser Extension (Javascript)
    • Again, manually use Websocket functions to communicate with server

Challenges we ran into

Since we challenged ourselves to use no pre-existing frameworks, we learned how to implement everything ourselves from near scratch. Most of our code exists within the standard library of Python, and entirely within the standard library of Javascript. This, however, does come at a cost of being much more complicated to implement.

Additionally, working with ffmpeg proved to be a big challenge, as its associated Python library acts merely as a wrapper around its CLI counterpart and does not give useful error messages whatsoever.

Accomplishments that we're proud of

Our core logic does work, and we are able to scan videos to detect flashing lights.

What we learned

  1. How videos format/compress their data
  2. How to work with individual frames + scaling with time & FPS
  3. How to consistently setup Ngrok API hosting
  4. How to work with raw Websockets to allow a client and server to persistently communicate over a stream on the internet

What's next for FFL: Heck Flashing Lights

Because all of our core logic was written in Python as a mere prototype, many improvements can be made. First of all, Python's frame parsing via OpenCV turned out to be quite a bottleneck in terms of speed. Additionally, because of the numerous dependencies associated with Python, all of the core logic necessarily had to be abstracted away to a remote server.

Both of these problems can be fixed by switching to a lower-level language like C or Rust. Not only will these languages provide an immense upgrade in terms of processing speed, but they also come with the added benefit of compiling to Web Assembly.

Web Assembly, or wasm for short, allows programs to be compiled directly into an instruction set runnable in browsers. This allows for Javascript files to interact with and run functions directly written in C/Rust, giving the speed of low-level and the frontend compatibility of JS!

Both of these are plans for the near future, as we all plan on seeing this project through.

Built With

Share this project:

Updates