Gestura

VoiceBridge AI helps speech-impaired people communicate using gestures. Detects gestures through the camera, interprets them with Gemini AI, and converts them into clear speech and text instantly.

Comment

Inspiration

We were inspired by the everyday communication challenges faced by speech-impaired individuals, especially in urgent situations where typing or pre-saved phrases are too slow. We wanted to create a simple, real-time solution that gives users a faster and more independent way to express their needs using gestures.

What it does

Gestura is an AI-powered communication assistant that helps speech-impaired users communicate through hand gestures. It detects gestures through a camera, interprets them using AI, and instantly converts them into clear text and spoken sentences. The platform also includes a gesture guide for learning common gestures and an emergency alert feature to quickly notify caregivers.

How we built it

We built Gestura as a web application using a modern frontend and backend architecture. The interface was designed to be highly accessible, with large buttons, clear text, and an interactive neon-themed layout. For gesture recognition, we used computer vision tools to capture and identify hand movements from the camera feed. The detected gesture is then processed through Gemini AI to generate a natural and meaningful sentence, which is displayed as text and spoken aloud using text-to-speech. The backend is designed to be deployable on Google Cloud for scalable real-time usage.

Challenges we ran into

One of the biggest challenges was making gesture recognition reliable under different lighting conditions and camera angles. Another challenge was ensuring the response felt real-time, since the experience needed to be smooth and natural for users. We also had to think carefully about accessibility and design an interface that is simple, fast, and easy to understand during stressful situations.

Accomplishments that we're proud of

We are proud of creating a solution that combines social impact with practical AI. Gestura is more than a technical demo—it is an assistive platform that can genuinely help speech-impaired individuals communicate more independently. We are also proud of integrating gesture recognition, AI interpretation, voice output, and emergency assistance into one unified application.

What we learned

Through this project, we learned how multimodal AI can be used beyond simple chat applications to solve real-world accessibility problems. We also gained experience in combining computer vision, AI-generated language, voice output, and cloud deployment into a single user-focused product. Most importantly, we learned that designing for accessibility requires both technical thinking and empathy.

What's next for Gestura

Our next step is to improve gesture recognition accuracy and expand support for a larger set of gestures, including more complete sign language patterns. We also plan to add multilingual voice output, caregiver notification through SMS or app alerts, and mobile support so Gestura can be used anywhere. In the long term, we want Gestura to become a reliable everyday communication companion for speech-impaired users.

Built With

Share this project:

Updates