PocketSign

Real-time AI that understands sign language and speaks it aloud — all in your browser.

Comment

Inspiration

We were inspired by the global challenge faced by over 400 million Deaf and Hard-of-Hearing people. Sign language is beautiful — but often misunderstood. We imagined a future where anyone could communicate using just hand gestures, and be heard instantly in any language, without installing any app or relying on the internet.

That’s where PocketSign was born — a tool that gives your hands a voice, accessible to anyone, anywhere, through just a browser.

What it does

PocketSign is a real-time sign language recognition system that:

📷 Uses your webcam to detect hand gestures

✋ Recognizes predefined sign language gestures

📝 Translates the gesture into on-screen captions

🔊 Speaks it aloud using the Web Speech API

🌐 (Optional) Translates output to other languages

And all of it happens right in the browser — no backend, no installs, no internet dependency.

How we built it

We used:

MediaPipe Hands for tracking 21 hand landmarks in real-time

Custom gesture logic in JavaScript to map finger positions to gestures

Web Speech API to speak the translated phrase

HTML/CSS + JS for a beautiful, responsive, and accessible UI

(Optional) LibreTranslate to support captions in multiple languages

We kept everything frontend-only to ensure privacy, speed, and offline usability.

Challenges we ran into

Calibrating gestures so they work reliably across different lighting, hand sizes, and angles

Designing a logic that is both fast and understandable for multiple signs

Ensuring accessibility with minimal distractions for users with different abilities

Getting Web Speech API to sound natural across browsers

Accomplishments that we're proud of

Built a fully functional sign-language translator in the browser — no server required

Created a peaceful, futuristic UI that feels intuitive and inclusive

Made something that could immediately help millions, especially in emergencies

Pushed the boundaries of AI + accessibility with no infrastructure cost

What we learned

How to use MediaPipe effectively for landmark detection

Gesture recognition logic using spatial coordinate mapping

Integrating Web Speech into UI events

Designing for accessibility and global users

Keeping frontend projects fast and functional without a backend

What's next for PocketSign

🎓 Train a real ML model using user-recorded gestures for more flexible recognition

✋ Support dynamic gestures and two-handed signs

🌐 Translate captions to 100+ languages using on-device translation

🤝 Open a community gesture training hub

📱 Release a mobile PWA version that works offline

We want PocketSign to become a universal translator for sign language, free and open for the world.

Built With

Share this project:

Updates