Inspiration
Communication is the foundation of human connection. yet millions of Deaf and hard-of-hearing individuals face daily barriers to being understood. We wanted to build something that doesn’t just translate gestures, but truly gives a voice to sign language, fostering inclusion and understanding everywhere.
What it does
Speak Bridge uses AI-powered computer vision to recognize American Sign Language signs and gestures to instantly convert them into spoken words and text in real time. The app creates seamless communication between ASL users and non-signers. Bridging communities through accessibility and technology.
How we built it
We used MediaPipe Hands for real-time hand landmark detection, paired with a Python FastAPI backend that sends frames to NeuralSeek who fed frames into a LLM for intelligent gesture classification. The React frontend streams camera input, displays sign detection results, and plays generated speech using a text-to-speech engine.
Challenges we ran into
Getting consistent hand tracking across different lighting and camera angles, Managing performance while streaming video frames to the backend, Designing prompts for NeuralSeek that produce accurate, low-latency classifications, Limited time to add more data and train for more accuracy, Handling real-time speech generation without lag.
Accomplishments that we're proud of
Achieving real-time ASL-to-speech conversion through efficient frame gating and batching, Building a full-stack pipeline that integrates AI, computer vision, and speech synthesis, Creating an inclusive product that highlights technology’s power to break communication barriers.
What we learned
We learned how to stream and process live video data, tune LLM prompts for multimodal reasoning, and design accessible, human-centered interfaces. Most importantly, we learned how to work in teams when developing a product.
What's next for Speak Bridge
We plan to fine tune our LLM for more consistent and accurate gesture recognition, expanding our database of gestures, and making the app more accessible.
Log in or sign up for Devpost to join the conversation.