Clip It

Clip It's home page
Highlights of the given video URL!
An annotated graph of part of a sample VOD. The orange indicates message frequency; red is the rolling median; and magenta is the difference

Inspiration

We are both avid esports fans, and the League of Legends world championship is happening now! This inspired us to create Clip It, a tool that can programmatically create highlight reels of Twitch streams in a matter of seconds.

What it does

Clip It allows you to find the highlights of your favorite streams. Given the URL for a video, it finds the highlights by tracking two key metrics: 1) Local spikes in chat message frequency (i.e. the equivalent of volume level in Twitch chat). We keep a rolling median of the number of messages sent in consecutive 30-second windows; and window that is significantly higher than the local median could be of interest! 2) Sentiment magnitude of the chat messages. To detect this, we used Google Cloud Platform's natural language processing-based sentiment analysis API, which returns sentiment score (i.e. positive or negative) and sentiment magnitude (how positive or negative). We focused on just sentiment magnitude, as that will pick out an interesting and wide variety of events, from controversial to exciting moments! Once again, we track local spikes in sentiment magnitude compared to the rolling median to find clips of interest.

We then look at the product of the differences between each metric and its rolling median--the clips with the highest value for this number therefore have large spike in both metrics. We return the timestamps with the highest values to make our highlight reel!

How we built it

We get Twitch chat data with the twitch-python API, and then analyze it with GCP's NLP API as well as our own statistical algorithm in Python. We host the backend server with Python + Flask. The frontend is in HTML + Javascript, connected to backend with Socket IO.

Challenges we ran into

-Integrating the front- and back-end proved difficult. -Finding the right way to track interesting events was a difficult choice to make.

Accomplishments that we're proud of

Implementing NLP was a cool accomplishment, as neither of us had tried using it before! We're also pleased with how closely the statistical analysis can track interesting moments of the stream--watching our first set of highlights was really gratifying.