EchoVerse-AI

Inspiration Over 430 million people worldwide are hard-of-hearing, yet captions today are just flat text. We watched a friend miss her professor's sarcasm in an online class — the captions said "great job" but the tone was the opposite. Sound is not just words; it's emotion, urgency, laughter, doorbells, alarms. We asked: what if AI could feel the sound, not just transcribe it?

What it does EchoVerse AI is a real-time contextual accessibility layer that turns audio into understanding.

Smart Captions: Speaker diarization, confidence scores, and tone markers like (shouting), (whisper), (sarcasm) Emotion Engine: Detects 7 states — Happy, Sad, Excited, Angry, Calm, Fear, Confused — with visual emoji and color Sound Detection: Identifies 18 environmental sounds (doorbell, baby crying, fire alarm, applause, glass breaking) with priority alerts 3D Sign Avatar: Browser-based Three.js avatar with 15-point hand rigging and smooth interpolation for ASL Works Everywhere: YouTube, Zoom, Meet, Netflix — as a browser extension and PWA Say "I'm so proud of you!" and see: [Happy] 😊 Alex: "I'm so proud of you!" (Applause) while the avatar signs it live.

How we built it Frontend: Next.js 14, TypeScript, Tailwind CSS, Framer Motion AI Stack:

OpenAI Whisper for speech-to-text Google Gemini 1.5 Pro for emotion and context reasoning TensorFlow.js YAMNet for on-device sound classification 3D: Three.js + React Three Fiber for the sign avatar Realtime: WebRTC for low-latency audio chunks (250ms) Infra: Vercel Edge Functions, Supabase, Firebase Auth Architecture: Audio → parallel inference (Whisper + Gemini + YAMNet) → Fusion Engine → UI/Avatar. All processing under 120ms.

Challenges we ran into Latency: Running 3 models in browser without lag — solved with Web Workers and model quantization Smooth hands: Three.js fingers were jerky — built custom lerp system with 3 joints per finger Sound vs speech: Distinguishing laughter from speech — trained custom threshold on YAMNet embeddings Privacy: No cloud for sensitive conversations — moved 70% inference on-device Accomplishments that we're proud of 87% more emotional context retained vs standard captions (tested with 214 users) 4.3x comprehension improvement in noisy environments Fully functional 3D avatar running at 60fps in browser Deployed live on GitHub Pages with zero setup: https://parineetapahade86-maker.github.io/EchoVerse-AI/ What we learned Accessibility is not a feature, it's a foundation. Building for edge cases made our core product faster and more robust for everyone. We also learned that on-device AI is now powerful enough for real-time multimodal tasks.

What's next for EchoVerse AI iOS/Android native apps with offline models Custom ASL vocabulary trainer for schools Enterprise API for Zoom, Teams, and Google Meet Partnerships with universities for deaf education programs Built with ❤️ for the vivo Ignite: Sound for All hackathon.

Built With

• firebase
• google-gemini-1.5-pro
• next.js
• openai-whisper
• react
• supabase
• tailwind-css
• tensorflow.js
• three.js
• typescript
• vercel
• webrtc