Fluent Meet-Sign Lang ⇄ Speech Translator & VidCall Platform

Inspiration

We were inspired by the communication barriers faced by deaf and hard-of-hearing people in classrooms, medical appointments, public services, and video calls. We wanted to create a practical bridge between sign language and spoken language, making communication more accessible without requiring a human interpreter for every interaction.

What it does

Fluent Meet is a real-time sign language to speech translator and video calling platform.

It can:

• Detect sign gestures through a webcam.
• Convert signs into text and spoken audio.
• Combine detected signs into natural sentences.
• Prevent repeated words when a gesture is held.
• Convert speech into live captions.
• Support English, Hindi, Marathi, and Spanish phrase mappings.
• Enable accessible video calls using Stream Video and Chat.

How we built it

We built the frontend with React and Vite. It captures webcam frames and sends them to a FastAPI inference service powered by PyTorch.

The AI model uses a compact DETR architecture to detect sign labels, confidence scores, and bounding boxes. Since individual frame predictions can be noisy, we added a temporal processing layer. A sign must receive at least 3 votes within a 5-frame window before it is accepted.

After detection, a gesture de-duplication system ensures that holding one sign does not produce repeated words. A sentence builder groups words until the user pauses or reaches the maximum sentence length. The resulting sentence is translated and spoken using the browser's Web Speech API.

Speech recognition provides live captions in the opposite direction, while Stream powers real-time video calling and chat.

Challenges we ran into

The biggest challenge was reducing flickering and false detections. A model may produce different labels across consecutive frames, even when the signer is making the same gesture. Temporal voting, confidence filtering, release detection, and pause-based sentence construction helped make the output more stable.

Running deep-learning inference on CPU infrastructure was another challenge. We had to keep the model compact while maintaining useful accuracy and reasonable response times.

We also dealt with camera lighting, browser permissions, audio autoplay restrictions, CORS configuration, deployment limits, and unreliable network conditions. To improve reliability, we added local development fallbacks and tested the application in different environments.

Accomplishments that we're proud of

We are proud to have built an end-to-end communication experience rather than just a gesture classifier. Fluent Meet connects computer vision, temporal processing, speech synthesis, live captions, multilingual translation, and video calling in one application.

We are especially proud of the temporal translation pipeline, which transforms noisy frame-level detections into stable words and complete sentences. We also built the system with browser-native speech tools, reducing the need for paid external APIs.

What we learned

We learned that real-time sign language translation requires more than detecting objects in images. A usable system must understand timing, repeated gestures, pauses, sentence boundaries, and the communication needs of both participants.

We also learned how to deploy and connect a React frontend, FastAPI service, PyTorch model, and real-time communication platform. Most importantly, we learned that accessibility should influence the entire product workflow, not be added as a feature at the end.

What's next for Fluent Meet-Sign Lang ⇄ Speech Translator & VidCall Platform

Our next goal is to expand beyond the current static sign vocabulary and support a larger Indian Sign Language dataset. We plan to add landmark-based temporal modeling for continuous and moving signs, improve sentence translation, and evaluate performance across different lighting conditions, backgrounds, and signing styles.

We also want to improve multilingual support, optimize inference latency, and make Fluent Meet more reliable for classrooms, healthcare, customer service, and remote collaboration.

Built With

• accessibility
• detr
• fastapi
• javascript
• multilingual
• opencv
• python
• pytorch
• react
• real-time
• rest
• tailwind
• text-to-speech
• vite