SceneSpeak

Inspiration

Video is hard to follow when you can't see it. People who are blind or have low vision miss what is happening on screen, and TV apps rarely help. We wanted a TV experience where short clips come with spoken narration describing the scene, so anyone can follow along from the couch with just a remote.

What it does

SceneSpeak is a TV app for Android TV and Amazon devices. Users browse a list of video clips with the remote, play them, and hear a narration track that describes what is happening on screen. Narration can be cycled between modes, and the whole interface works with a D-pad, with no touch needed.

How we built it

• A React Native / Expo app targeting TV (react-native-tvos), in a monorepo with a separate Amazon Vega app sharing UI code.
• react-tv-space-navigation for focus and remote-control navigation.
• react-native-video for playback, with a video overlay and a narration hook that syncs audio to the clip.
• A pipeline that prepares clips and narration. [Describe: e.g. generates descriptions and text-to-speech audio.]
• We forked the open-source react-native-multi-tv-app-sample and extended it with narration, clip handling, and playback controls.

Challenges we ran into

Getting a modern Expo SDK 54 TV build running on Windows was the hardest part. We hit entry-file problems in the monorepo, mismatched native package versions, Reanimated 4 requiring the New Architecture, and antivirus blocking the Android NDK compiler during native builds. We solved it by fixing the entry point, aligning dependencies, enabling the New Architecture, and moving the native build to Expo EAS Build.

Accomplishments that we're proud of

• A fully remote-navigable TV experience with narrated clips.
• One codebase approach shared across Android TV and Amazon Vega.
• Getting a complex TV toolchain working end to end.
• [Add: anything specific, e.g. narration sync quality.]

What we learned

• TV UX is its own discipline: focus management matters more than layout.
• Native tooling versions (Expo, React Native, Reanimated, NDK) must be kept aligned.
• Accessibility features like narration need to be designed in from the start.

What's next for SceneSpeak

• Automatic narration generation for any video.
• More languages and voices.
• Adjustable narration speed and detail level.
• Support for more TV platforms and a larger clip library.

Built With

• amazon-vega
• android-tv
• expo.io
• polly
• python
• react-native
• react-native-reanimated
• react-native-tvos
• react-native-video
• react-navigation
• react-tv-space-navigation
• text-to-speech
• typescript
• yarn