SignTogether : AI-powered accessible video meetings for all

SignTogether makes video calls accessible to people with hearing loss. AI translates sign language to text and speech to captions in real time. No interpreters needed. Just instant inclusion.

Comment

Inspiration

Over 430 million people worldwide have disabling hearing loss, yet they're systematically excluded from video calls—the primary mode of communication in our digital world. I witnessed this barrier firsthand and realized that existing video platforms treat accessibility as an afterthought, forcing deaf users to rely on interpreters or simply miss out on conversations entirely.

I built SignTogether because communication is a fundamental right, not a privilege. The inspiration came from seeing how AI has revolutionized language translation, and asking: why can't we do the same for sign language and gesture-based communication? With modern AI capabilities, we can finally bridge the gap between deaf and hearing users in real-time, making every video call truly inclusive.

What it does

SignTogether is an AI-powered video meeting platform that enables seamless communication between deaf and hearing participants:

Real-time Gesture & Sign Language Detection: Using Google Gemini Vision AI, the platform analyzes video frames every 1.5 seconds to detect sign language gestures, hand movements, and common signs like "thumbs up," "peace," "wave," and active signing. Live Speech-to-Text Captions: Deepgram's Nova-2 model transcribes spoken words into text with 95%+ accuracy, displaying captions in real-time with confidence scores. Unified Caption Feed: Both gesture detections and speech transcriptions appear in a single live caption stream, color-coded to distinguish between signed and spoken communication. AI Meeting Summaries: After meetings, Anthropic Claude generates intelligent summaries with key discussion points and action items. High-Quality Video: LiveKit powers the video infrastructure, ensuring low-latency, high-quality video calls. The result? Deaf participants can sign naturally while hearing participants speak, and everyone sees the same conversation in text—no interpreters, no delays, no barriers.

How we built it

Frontend Architecture: Next.js 14 with App Router for modern React development TypeScript for type safety and better developer experience TailwindCSS for responsive, accessible UI design React hooks for real-time state management and video processing

AI Integration: Google Gemini Vision API: Processes video frames captured via Canvas API to detect gestures and sign language with confidence scoring Deepgram Nova-2 API: Transcribes audio streams captured via WebRTC with smart formatting and punctuation Anthropic Claude 3.5 Sonnet: Generates meeting summaries and cleans up transcripts

Real-time Processing Pipeline: Camera stream captured via getUserMedia() Video frames extracted to canvas every 1.5 seconds Frames converted to base64 JPEG and sent to Gemini Vision API Audio captured via MediaRecorder and sent to Deepgram in 3-second chunks Results displayed in unified caption feed with timestamps

Backend & Database: Next.js API routes for serverless functions Prisma ORM with SQLite for data persistence LiveKit Server SDK for video token generation Graceful error handling to work without database dependencies

Challenges we ran into

  1. Real-time Performance Optimization Balancing AI processing speed with accuracy required careful tuning. We experimented with frame capture intervals (3s → 1.5s), image compression (70% → 50%), and confidence thresholds (40% → 25%) to find the sweet spot between responsiveness and accuracy.

Accomplishments that we're proud of

✅ Built a fully functional accessibility platform in 36 hours with real-time AI processing ✅ Achieved sub-2-second latency for both speech and gesture recognition ✅ Successfully integrated 4 different AI APIs (Gemini, Deepgram, Claude, LiveKit) into a cohesive experience ✅ 90%+ gesture detection accuracy and 95%+ speech transcription accuracy in testing ✅ Created an intuitive interface that requires zero training—just join and start communicating ✅ Solved real-world accessibility challenges that affect 430+ million people worldwide ✅ Made the app work without database dependencies, ensuring it runs anywhere

Most importantly, we're proud that SignTogether isn't just a tech demo—it's a production-ready solution that could genuinely improve lives.

What we learned

Technical Skills: How to architect real-time AI applications with multiple concurrent processing streams Best practices for integrating vision and speech AI models in production environments The importance of graceful degradation and comprehensive error handling in accessibility tools How to optimize AI API calls for both cost and performance Deep understanding of WebRTC, Canvas API, and browser media APIs React performance optimization techniques for real-time applications

What's next for SignTogether

Immediate Roadmap (Next 3 months):

Expand sign language support to ASL, BSL, ISL, and other regional sign languages Implement mobile apps for iOS and Android with optimized gesture detection Add real-time translation between different sign languages Build a gesture vocabulary training system for custom workplace or family gestures

Ultimate Goal: Make SignTogether so ubiquitous that accessibility in video calls isn't a special feature—it's just how video calls work. Because everyone deserves to be part of the conversation.

Built With

Share this project:

Updates