Watch & Learn

Turn any favorite video into an interactive English lesson with clickable subtitles and instant explanations.

Inspiration

I was inspired by language-learning apps that use videos and interactive subtitles. I liked the idea, but I wanted something simpler, more personal, and free to use locally.

The project also came from a real need at home. My four-and-a-half-year-old daughter loves Frozen and has watched it twenty or thirty times. She understands the story, but she still misses or repeats some lines incorrectly.

I considered printing the entire script and translating it line by line, but following a printed script while watching the movie was inconvenient. I wanted us to watch together, follow the subtitles on the same screen, and quickly look up anything unfamiliar.

That became Watch & Learn.

What It Does

Users can upload a video and its English subtitle file. The app displays the complete script and highlights the current line during playback.

When the user clicks a word, the video pauses and a Chinese explanation appears. Closing the explanation resumes playback. Users can also translate the complete sentence and adjust the subtitle timing.

Everything runs locally, so videos and subtitles stay on the user’s computer.

How I Built It

I built Watch & Learn as a web application using Next.js, TypeScript, HTML, CSS, and JavaScript, and deployed it with Vercel.

The app supports both SRT and VTT subtitles. Videos and subtitle files remain on the user’s device and are stored locally in the browser, so personal media does not need to be uploaded to a server.

For translation, the app sends only the selected word or subtitle sentence to Gemini’s free API. Word meanings and sentence translations are cached in the browser, making repeated requests much faster while reducing API usage.

I also built custom subtitle synchronization, automatic highlighting, sentence replay, click-to-seek playback, and an interactive copyright-friendly demo.

Challenges

The biggest challenge was balancing translation quality, response speed, and cost.

AI can provide much richer explanations than a traditional dictionary, especially for slang and expressions. However, longer prompts and answers made frequent lookups feel slow. Because the current version uses Gemini’s free API tier, it can also encounter temporary availability and usage limitations.

I improved the experience by creating shorter, task-specific prompts: word lookups return only concise meanings, while sentence requests focus on a clear translation. I also added browser caching, so repeated words and sentences appear almost instantly without making another API request.

Subtitle synchronization was another challenge because subtitle files do not always align perfectly with their videos. I added precise timing controls that move subtitles earlier or later in 0.05-second increments.

Finally, I had to preserve a smooth learning experience while users clicked words, replayed sentences, scrolled through the transcript, and moved between different lines. The app now pauses during translation, resumes afterward, highlights the current subtitle, and lets users start playback from any sentence.

What I Learned

I learned how to connect video playback, timed subtitles, browser storage, and cloud AI in one cohesive application. I also learned how small design decisions—such as pausing during translation, caching repeated requests, and allowing precise subtitle adjustments—can make a learning experience feel much smoother.

More importantly, I learned that language learning does not have to begin with a textbook. It can begin with a story, character, or movie that someone already loves.

Watch & Learn turns familiar videos into opportunities to listen more carefully, understand real expressions, repeat difficult lines, and enjoy the process of learning English.