HandGestury

• 

  Homepage

• 

  Workflow

• 

  Launch Translator page

• 

  Live Translation

Inspiration

Communication barriers between the deaf and hearing communities remain a significant challenge. We wanted to create a solution that leverages AI and computer vision to make communication more accessible and inclusive. This inspired us to build HandGestury, a real-time sign language translator that converts hand gestures into text directly in the browser.

What it does

HandGestury is an AI-powered web application that recognizes sign language gestures using a webcam and translates them into text in real time. Users can perform hand gestures in front of their camera, and the application detects and interprets them instantly, enabling smoother communication between sign language users and non-signers.

How we built it

We built HandGestury using React, TypeScript, and Vite for the frontend. The user interface was designed with Tailwind CSS and shadcn/ui to provide a modern and responsive experience. For gesture detection, we integrated MediaPipe Hands, which tracks hand landmarks in real time. We then processed these landmarks to identify specific gestures and map them to corresponding text outputs. The application runs entirely in the browser, making it lightweight and easily accessible.

Challenges we ran into

One of the biggest challenges was achieving reliable gesture recognition across different lighting conditions, hand positions, and camera angles. We also faced difficulties minimizing false detections while maintaining real-time performance. Optimizing the hand-tracking pipeline and ensuring a smooth user experience across devices required extensive testing and fine-tuning.

Accomplishments that we're proud of

We are proud of building a fully functional real-time sign language translator that runs directly in the browser without requiring special hardware. Successfully integrating AI-powered hand tracking, delivering low-latency gesture recognition, and creating an accessible user experience were major milestones for our team.

What we learned

Through this project, we gained practical experience with computer vision, MediaPipe hand tracking, real-time data processing, and modern web development technologies. We also learned the importance of accessibility-focused design and how AI can be used to solve real-world social challenges.

What's next for HandGestury

We plan to expand the gesture vocabulary, support complete sentence formation, and improve recognition accuracy using machine learning models. Future versions will include text-to-speech functionality, support for multiple sign languages, and enhanced accessibility features to make communication even more seamless and inclusive.

Built With

• google-mediapipe
• html5
• javascript
• react
• tensorflow