Live Audio Captioner

Real-time captions, translation, and spoken dubbing for Fire TV, powered by Amazon AI services to make video more accessible across language barriers.

Comment

Inspiration

Video content is increasingly global, but language remains a barrier. Educational videos are a particularly important example: useful content may exist, but viewers cannot fully benefit from it if they do not understand the spoken language.

Live Audio Captioner was created to explore how Fire TV could make video more accessible by adding real-time captions, translation, and spoken dubbing without interrupting the viewing experience.

What it does

Live Audio Captioner adds AI-powered language features directly to video playback on Fire TV.

Using the Fire TV remote, the viewer can instantly switch among:

  • Captions — converts the program audio into readable text in real time.
  • Translate — translates the recognized speech between English and Spanish.
  • Dub — converts the translated text into spoken audio while reducing the volume of the original program.
  • Hidden — returns to normal viewing with the caption overlay removed.

The viewer can also switch the language direction between English → Spanish and Spanish → English with the Select button.

How I built it

The application was built in Kotlin for Android TV / Fire TV using Android Studio and ExoPlayer.

Program audio is captured directly from the ExoPlayer audio pipeline and processed through Amazon Web Services:

  • Amazon Transcribe Streaming provides real-time speech recognition.
  • Amazon Translate translates recognized speech between English and Spanish.
  • Amazon Polly generates the spoken translated voice used in Dub mode.
  • Amazon Cognito Identity provides temporary AWS credentials without embedding long-term AWS access keys in the application.

The Fire TV remote provides direct controls for Captions, Translate, Dub, Hidden mode, language-direction switching, and Play/Pause.

Challenges

One of the biggest challenges was capturing the program audio itself rather than relying on a microphone. The solution was to capture PCM audio directly from ExoPlayer and stream that audio to Amazon Transcribe.

Real-time dubbing presented another challenge. Polly speech segments initially could overlap or continue playing after the viewer changed modes. I added an audio queue so translated speech plays sequentially, along with safeguards that immediately stop and clear queued Polly audio when leaving Dub mode or switching language direction.

Testing also revealed differences between the Android TV emulator and a physical Fire TV device. Final validation was therefore performed on a Fire TV Stick 4K Max.

Accomplishments

The completed prototype demonstrates:

  • Real-time captions from program audio
  • Bidirectional English/Spanish translation
  • Spoken AI dubbing
  • Direct Fire TV remote interaction
  • Automatic reduction of original audio during dubbing
  • Clean switching between modes without leaving video playback
  • Operation on an actual Fire TV Stick 4K Max

What I learned

This project required integrating streaming audio processing, cloud speech recognition, translation, text-to-speech, asynchronous audio playback, and Fire TV remote interaction into one continuous viewing experience.

It also reinforced the importance of testing on real Fire TV hardware rather than relying entirely on an emulator.

What's next

Live Audio Captioner could evolve from a multilingual viewing tool into an interactive AI learning platform. Additional languages and selectable video sources would extend the current captioning, translation, and dubbing experience. A production version could continue to scale these capabilities through Amazon Transcribe, Amazon Translate, and Amazon Polly.

A major next step would be conversational AI powered by Amazon Bedrock. Instead of passively watching a recorded lecture, documentary, podcast, or other video, viewers could ask questions about what they are watching, request simpler explanations, generate summaries or quizzes, compare concepts, and return to the portion of the video where a topic was discussed.

Alexa+ could provide a natural voice interface for this experience on Fire TV and across Alexa-enabled devices, allowing viewers to ask questions, request explanations, summarize content, or navigate to specific topics simply by speaking.

For education, this could transform a recorded lecture into an interactive tutoring experience. With the instructor's permission, future versions could create a clearly identified AI teaching companion grounded in that instructor's lectures, course materials, explanations, and teaching approach. Students could continue asking questions and receiving instruction long after the original recording ends.

Accessibility is another important opportunity. Real-time captions, translation, and AI-assisted viewing could help educational institutions make video content more accessible across both disability and language barriers and support broader digital-accessibility goals.

Although education is a primary use case, the same architecture could also extend to news, sports, documentaries, podcasts, entertainment, and other video content.

The long-term vision is to move from simply watching video to interacting with it: watch, caption, translate, dub, understand, ask, and learn.

Built With

Share this project:

Updates

Submission history