There are many people who need subtitles for their videos. These videos are not only limited to movies or TV, but any sort of media that has audio has a potential benefit from subtitles. Home videos, podcasts, lectures, and of course the hundreds of ahem perfectly legal movies on your hard drive. AI technology has long been advanced enough to do this, but so far only YouTube has made an effort. We thought to give it a try as well.
What it does
The user simply provides the interface with a video file, and the application then generates synced subtitles and provides the link for download. Note: We have to wait for IBM Watson to process the information, and so even though we provide the link immediately it will probably not work if you click on it too early. Please wait a little.
How we built it
IBM Watson provides a Speech to Text API that we can input audio clips and get transcripts with approximate timings. Therefore, we first rip the audio from the video using FFmpeg, send it to Watson and wait for a response, and then format the data into a subtitle file (SRT).
Challenges we ran into
Arman I had started using Node.js to write an app but it quickly became more and more complex and messy. Furthermore, since I was not very experienced with asynchronous programming I fell into "callback hell". Later I rewrote the whole app from scratch using a simple approach, eliminating more than half of the code and making it much cleaner.
Some challenges were Watson's processing speed - incredibly slow, you would have to wait the entire length of the video you want for it to be finished. I had played a little with parallel requests to the API which greatly improved the speed, but it became hard to manage the code, and in interest of time I abandoned that.
Accomplishments that we're proud of
Becoming comfortable with asynchronous programming and node callbacks. Finishing the project within 36 hours, so we have a working subtitle generator. Learned when to rm -rf * and start again (or just slow down)
What we learned
Node. Callbacks. Promises. APIs.
What's next for Watson AutoSubber
I didn't have enough time during this hackathon, but I want to implement parallel requests which will greatly improve the performance.