AI-based video subtitle translator

Stop missing out on movies, because you don't understand the language. With the "AI-based video subtitle translator" you can hook into any* web-based video player and have your subtitles translated!

Comment

Inspiration

Giving people the tools to understand the world around them is key to education. There are a lot of video resources online that have only one (maximum two) subtitles available, and the audio is usually in a single language. It costs money and time to have more subtitles available. With this solution, we would like to help on this issue.

What it does

The solution consists of 3 parts:

  1. a NPM module, so websites with video players can just use it and enable AI translations for the videos.
  2. a webapp that showcases how to use the NPM module on an open-source and a commercial video player
  3. a Chrome extension that can hook into ANY webpage with a video element and a subtitle track enabled. (with limitations).

At the end of the day with either solution, you just need one single subtitle track (preferably English), and the solution can translate it with the use of AI.

How we built it

First I built an NPM package that helps easily hook into video players. Then I realized that probably it would not qualify for the challenge, so I created a demo web app, and while building that the idea of a "universal extension" just popped.

Challenges we ran into

  1. It seems the downloadprogress events are not working in the latest Canary. (or based on the doc I wasn't able to figure it out)
  2. Packaging issues while building a webapp.
  3. Otherwise the integration with the AI API was pretty smooth.
  4. Time

Accomplishments that we're proud of

Unfortunately, I was short on time, but I'm really happy with the result I achieved within my 20-hour mini hackathon. The code is pretty half-baked, I focused on delivering results, and I reached the goals I set for myself.

What we learned

I'm not a web developer, so for me building web stuff from scratch is always challenging. I really like where AI is moving, so I wanted to give a try to this challenge anyway. I've learned about JS modules, setting up a TS project from scratch, building and using an NPM module, and building a Chrome extension. The built-in AI APIs within Chrome are pretty cool, and it is nice that any of the requests will not leave the user's machine.

What's next for AI-based video subtitle translator

  • The project has some limitations and known issues, I would like to fix those.
  • After that, I would like to work on supporting more video players.
  • Use the Language Detection API, so customers have to configure less when using the NPM module or the extension
  • The Chrome extension was really rushed out, it needs some extra care.
  • Provide alternative ways for translation rendering. (e.g. adding new TextTracks to the video elements, instead of having my own rendering)
  • I would like to introduce some performance measurements. e.g. if the AI is generating the subtitles slow on the fly, we could do it upfront.
Share this project:

Updates