BridgeTalk

Inspiration

Social media and video calls are a big part of communication today, but people who can't speak or hear are often left out because most others don’t understand sign language. That’s why We wanted to give those people a voice in that virtuel world , by building a tool that helps both sides talk to each other in real time, by translating sign language to voice and voice to sign language during live video calls.

What it does

   BridgeTalk is a real-time communication tool that translates sign language into speech and spoken voice into sign language. It enables deaf, mute, and hearing individuals to engage in natural, two-way conversations during video calls , without the need for an interpreter.

How we built it

Video Call Setup (WebRTC): We used WebRTC to create a live video call between two users.
Sign Language Detection (MediaPipe + OpenCV): On the deaf user’s side, we used the webcam to capture hand movements. MediaPipe and OpenCV helped us track and detect the shape and position of the hands in real time.
Gesture Recognition (PyTorch +** NumPy**): We used a simple PyTorch model to recognize specific hand gestures (like "Hello" or "Help") from the camera feed. NumPy helped us process the hand landmark data.
Text-to-Speech (TTS): After detecting the sign, we turned it into text, then used a text-to-speech tool to convert that text into voice, so the hearing user could hear the message.
Speech Recognition (STT): When the hearing person responds, we used speech-to-text (STT) to convert their voice into text.
Show Sign Language (Images or Animation): We showed the matching sign (as an image or animation) on the screen for the deaf user to understand what the other person said.
Real-Time Interaction: All parts run together in real time, allowing both users to “talk” to each other, even if one uses voice and the other uses sign language.

Challenges we ran into

Collecting and preprocessing high-quality sign language data for accurate detection.
Minimizing latency to ensure smooth and real-time translations during video calls.
Ensuring consistent detection across different conditions.

Accomplishments that we're proud of

Successfully developed a working prototype that enables two-way real-time translation in video calls.
Designed an inclusive and accessible experience for deaf, mute, and hearing users.
Optimized translation speed to ensure smooth, uninterrupted communication.

What we learned
The importance of teamwork in combining AI, project designing, and communication technologies.
A deeper understanding of real-time media streaming and its challenges.
How meaningful it can be to build technology that makes a social impact.

What's next for BridgeTalk

Expanding the sign language dataset to support more gestures, dialects, and regional variations.
Adding group video call support and multi-language translation capabilities.
Partnering with accessibility organizations and educational institutions to bring BridgeTalk to those who need it most.