Hearmeout

application

Project Description

Problem Statement

For Deaf and hard-of-hearing people, everyday conversations with people who don't know sign language are a constant barrier. They sign - and the person in front of them can't understand. When that person replies out loud, they can't always read lips. Existing accessibility tools usually solve only one half of this: they translate signs to text, but don't cover the spoken reply. A real conversation is two-way, and it needs a two-way bridge - especially when there is no interpreter or common language available.

Solution Overview

HearMe Out is an AI-powered, two-way communication bridge that runs entirely in the browser. The user signs toward their camera and HearMe Out recognizes the gesture with an on-device machine-learning model (Google MediaPipe hand tracking), then speaks the words out loud in a natural neural voice (Microsoft Edge TTS). When the other person replies, their speech is transcribed live into captions on screen. No install, no accounts, no API keys, and no video ever leaves the device.

Key Features

Sign recognition (AI): 7 hand signs (Hello, Yes, No, I love you, Victory, Excuse me, Stop) recognized on-device via 21 hand landmarks, with a hold-time + confidence threshold to reduce false positives.
Neural AI voice: every phrase is spoken by Microsoft Edge TTS (free, no API key) through a small serverless function that streams MP3 back.
Live captions: the other person's speech appears as real-time captions (Chrome / Edge) so the Deaf user can follow the reply.
Type to talk: a chat box for when signing isn't possible.
Quick-phrase buttons: one-tap phrases that always work, even if a sign is missed.
Mute, camera zoom, and auto-speak toggles for flexible use.
Privacy-first: gesture recognition runs on-device; no video is uploaded, no accounts, no keys.

Technologies Used

Machine Learning / Computer Vision: Google MediaPipe hand-tracking model (on-device, ~21 landmarks) for gesture recognition
Speech output: Microsoft Edge TTS neural voices via a Vercel serverless function (api/speak.ts)
Speech input: browser Web Speech API for live captions
Frontend: Vite + TypeScript + Tailwind CSS
Deployment: Vercel (HTTPS, free)

Target Users

Deaf and hard-of-hearing people who want to talk with anyone – family and friends who don't sign, teachers and classmates in classrooms, service and customer-facing situations, and anyone without a common language or an interpreter available.