EchoSign

A Fire TV accessibility app that translates sign language into spoken words, giving everyone a voice in the smart home. 100% serverless and private, with zero cloud latency.

Comment

Inspiration

Voice assistants revolutionized the smart home, turning science fiction into reality by letting us control our environments with just a few words. We wanted to ensure this incredible technology was fully inclusive for everyone.

While smart home hubs are built entirely around spoken commands, we realized the living room TV was the perfect canvas to build an accessibility bridge for non-verbal individuals. We envisioned a tool that transforms any Fire TV equipped with a standard USB webcam into an adaptive smart home hub, giving everyone an equal voice in their home without requiring complex authentication or bulky hardware.

We wanted to build an accessibility bridge that didn't require complex authentication, bulky hardware, or exposing vulnerable users to privacy risks. The living room TV was the perfect canvas. We envisioned a tool that transforms any Fire TV equipped with a standard USB webcam into an adaptive, accessible smart home hub.

What it does

EchoSign is a smart TV application that translates sign language and gestures into spoken words in real-time.

Instead of forcing users to build a native Alexa Skill (which introduces massive friction via account linking and authentications), EchoSign acts as an Acoustic Proxy. The user signs to the TV's webcam, and the TV simply speaks the translation out loud through its speakers. Any nearby Echo device or Google Home hears it and reacts naturally—exactly as if the user had spoken the words themselves.

For a completely silent experience, or for controlling devices in other rooms, EchoSign also features a Universal Webhook Architecture. A user can map a gesture combo to a webhook URL. When the sign is performed, EchoSign silently fires an HTTP POST request across the internet, acting as an invisible remote control to instantly trigger services like IFTTT or Home Assistant.

How we built it

EchoSign is built as a lightweight, highly-optimized React & TypeScript web application, designed specifically for the 10-foot Fire TV experience. It leverages the Amazon Web App Runtime, meaning it runs natively in the Fire TV Silk browser and utilizes standard WebRTC for the camera feed.

The core spatial tracking is powered by MediaPipe Hand Landmarker. Rather than relying on rigid, pre-programmed alphabets, we built a custom mathematical engine. When a user creates a new "combo," the app extracts a 3D feature vector (the mathematical shape of the hand) and saves it to the browser's LocalStorage. When the camera is live, it compares incoming frames against these saved vectors using a strict Euclidean distance threshold to find a match.

Voice synthesis is handled entirely offline via the native Browser SpeechSynthesis API, ensuring zero-latency text-to-speech.

Challenges we ran into

  1. Hypersensitivity: Initially, our gesture engine was evaluating shapes at 60 frames per second. This made the AI hyper-sensitive, instantly triggering commands if a user's hand briefly passed through a known shape while moving. We had to engineer a custom "Frame Throttle" and "Debounce" system. It now requires the user to hold the shape completely still for a specific number of frames before the vector math confirms the shape, filtering out all transitional movements.
  2. Privacy on a TV Camera: Users are rightfully cautious about putting cameras in their living rooms. We had to ensure absolute privacy. We architected the app to be 100% serverless. The computer vision math processes entirely on the local TV processor, meaning zero video frames are ever recorded or transmitted to the cloud.

Accomplishments that we're proud of

We are incredibly proud of the Custom Combo System. Instead of forcing a user to learn and memorize 50 unique hand poses for 50 different smart home commands, we allow them to build macro sequences. A user can reuse a small handful of comfortable signs and string them together (e.g., combining a peace sign and a thumbs up to mean "Turn on the lights"). This drastically lowers the learning curve and makes the app accessible to users with limited motor function.

What we learned

We learned the incredible power and capability of the modern web stack on consumer hardware. Proving that an inexpensive Fire TV stick can run complex, real-time 3D computer vision entirely inside the browser without relying on expensive cloud APIs was a massive breakthrough for us.

What's next for EchoSign

Our next step is packaging the web application for official deployment to the Amazon Developer Console. By wrapping the application in a native APK shell, users will be able to download EchoSign directly from the official Fire TV Appstore, bringing frictionless, private accessibility to living rooms everywhere.

Built With

Share this project:

Updates

Submission history