SunoSaathi

Inspiration

With 18 million deaf and hard-of-hearing people in India, everyday communications with the general hearing world remain broken. Most hearing people do not know ISL, and most deaf users cannot rely on English/Hindi text due to literacy gaps. Newer tools like captioning apps only translate speech to text; interpreter services for ISL are prohibitively expensive and inaccessible in rural areas. We wanted to develop something to give each deaf person an equal voice without waiting for interpreters, without needing literacy, and not depending on one language. That is how SunoSaathi was born-a multilingual AI-powered companion allowing deaf and hearing people to communicate naturally.

What it does

SunoSaathi allows real-time two-way communication between deaf and hearing users through: Speech → ISL avatar + subtitles ISL → text → speech Automatic translation of text into more than 12 Indian languages Safety filtering to prevent harmful/abusive content Works in any phone or browser without special hardware It's like having an instant, always-available AI ISL interpreter in your pocket.

How we built it

We designed SunoSaathi as a modular, fully AI-driven pipeline: Whisper STT for accurate speech-to-text. Language detection: fastText / LangDetect Clean text by using spaCy with regex and IndicBART. HuggingFace Toxicity Filter for safety IndicTrans2: English ↔ Indian languages Extraction of Keywords using KeyBERT & spaCy ISL Animation Pipeline using 3D Avatar rigged in Mixamo + Blender, Three.js + GLTFLoader to play .glb animations, JSON-based keyword → sign mapping, React frontend + FastAPI backend and Hosted on Render + Firebase Storage We focused on building a real-world pipeline that works on ordinary devices and does not require any proprietary hardware.

Challenges we ran into

Lack of publicly available ISL datasets for gloss or sign generation Grammar and dialects are very complex in Indian languages. Making the animations of the avatars appear natural and not robotic. Ensuring safety filtering happens before translation to prevent harmful content leakage. Dealing with noisy audio environments, typical in India Solution design that works for users having low literacy and low bandwidth