Apply machine learning to help gfycat build a semantic database of gifs.

What it does

As a minimal viable project, it detects laughter when user watches a video on youtube (all without user inputs) and laughs at the moment when there is laughter. This allows gyfcat to increase user engagement (a native app instead of just a website) as well provide a way to gather semantic data (map gifs to a particluar youtube video at a time segment). This is a positive feedback loop as more analysis can be on the youtube segment and therefore allow richer understanding of gifs and therefore create more signals for contextual engagements.

How I built it

Scraped a mini datasets of positive and negative samples of laughter, prototyped and tested with a neural network classifier (NN) as well as recurrent neural network classifier (RNN) with keras backed by tensorflow.

From there, built a simple flask server to serve traffic in production with the learnt model by 1) downloading the youtube link, 2) extracting its audio, 3) converting the audio mp3 to wav, and finally 4) segmenting wav into 2 second segments and apply classification to all the segments.

Build a frontend electron app to send youtube link to backend for analysis when it automatically detects a user is watching a youtube link and laugh with user.

Challenges I ran into

Trying to find an open dataset online but ending up having to gather my own dataset. Experimented with RNN but unfortunately not enough data as validation accuracy is lower than a simpler NN. Optimized for precision instead of accuracy. Not enough time to experiement with more data cleaning and gathering.

Having a stable production backend (configuration with nginx, ssh tunnels, etc.)

Accomplishments that I'm proud of

End to end machine learning application (from raw data, training, production deployment, and frontend app that uses it).

What I learned

ML is all about data and data.

What's next for gitcat

More contextual signals and build the platform to generate semantic tags for gifs and therefore build a valuable database of contextual gifs.

Built With

Share this project: