Inspiration
Oftentimes it takes quite a long time to find the right spot in longer videos, even when we know exactly what we are looking for. This is especially true with lectures and documentaries. With recent developments in audio transcription, we can now search many videos by their dialog content.
What it does
Scrub uses the captions generated (and sometimes hand-written) for YouTube videos to allow users to query locations in videos where a certain word or phrase is spoken. We are also looking into using the speech recognition capabilities of iOS 10 to allow us to transcribe personal videos as well.
How we built it
We built both an iOS mobile app and a React web app for this, both utilizing an XML-based API for YouTube to receive generated captions for videos. Embeddable YouTube players allow us to play the video at the given time stamp of a phrase when it happens. For the iOS personal video dialog search, we are investigating using the MediaPlayer and Speech frameworks to locate and transcribe video files.
Challenges we ran into
Unexpected issues with YouTube players due to changes in iOS 10, as well as several issues involving parsing XML and launching Herold to host the web app. Fairly smooth sailing otherwise, though!
Accomplishments that we're proud of
Being able to quickly show all instances of a word or phrase in an hour-long lecture video within seconds in a clean UI for both iOS and web.
What we learned
There's still a long way to go with speech-to-text, particularly in the face of noise, accents, and so forth. Additionally, some YouTube videos don't allow access to captioning via their API due to copyright issues...
What's next for Scrub
Getting a working implementation of personal video transcriptions and further optimizing the transcription process. Hopefully developing an Android version as well and a simpler Chrome extension to allow users to use Scrub while they are on YouTube itself.
Log in or sign up for Devpost to join the conversation.