Ever sit through a long and excruciating video like a lecture or documentary? Is 2x speed too slow for youtube? TL;DW
What it does
Just put in the link to the YouTube video you are watching, then wait as our Revlo and NLTK powered backend does natural language processing to give you the GIFs from GIPHY that best reflect the video!
How I built it
The webapp takes in a link to a youtube video. We download the youtube video with pytube and convert the video into audio mp3 with ffmpeg. We upload the audio to Revspeech API to transcribe the video. Then, we used NLTK (natural language toolkit) for python in order to process the text. We first perform "part of speech" tagging and frequency detection of different words in order to identify key words in the video. In addition, we we identify key words from the title of the video. We pool these key words together in order to search for gifs on GIPHY. We then return these results on the React/Redux frontend of our app.
Challenges I ran into
We experimented with different NLP algorithms to extract key words to search for gifs. One of which was RAKE keyword extraction. However, the algorithm relied on identifying uncommonly occurring words in the text, which did not line up well in finding relevant gifs.
tf-idf also did not work as well for our task because we had one document from the transcript rather than a library.
Accomplishments that I'm proud of
We are proud of accomplishing the goal we set out to do. We were able to independently create different parts of the backend and frontend (NLP, flask server, and react/redux) and unify them together in the project.
What I learned
We learned a lot about natural language processing and the applications it has with video. From the Rev API, we learned about how to handle large file transfer through multipart form data and to interface with API jobs.
What's next for TLDW
Summarizing into 7 gifs (just kidding). We've discussed some of the limitations and bottlenecks of our app with the Rev team, who have told us about a faster API or a streaming API. This would be very useful to reduce wait times because our use case does not need to prioritize accuracy so much. We're also looking into a ranking system for sourced GIFs to provide funnier, more specific GIFs.