SignScribe

SignScribe
Interpreting hand sign with caption
Extension overlay

We have always believed that communication is the most paramount aspect of human connection. Yet, millions of people who are deaf or mute face unnecessary barriers simply because they were born that way. It’s unfair that their voices are often unheard in a world designed for spoken communication.

This inspired us to build Signscribe—a tool that empowers individuals by translating sign language into text in real time, helping them communicate more freely in virtual spaces. Our mission is to make communication accessible for everyone, by _ Giving Power to Every Gesture _ .

Throughout the development of SignScribe, we gained valuable insights into several technical areas.

Machine Learning (ML) and AI: We explored computer vision models to recognize ASL signs, learning about the complexities of model training and fine-tuning.
Computer Vision: We applied YOLO11 for sign detection, experimenting with different architectures to improve accuracy.
Chrome Extension Development: We learned how Chrome extensions are structured and how to manage permissions and performance constraints with typescript.
Model Optimization: We realized how time-consuming and resource-intensive building and refining even a simple identification model can be.
Constraints: We faced limitations with processing power, video handling, and API access within Chrome’s environment, pushing us to optimize our code for efficiency.

How We Built It

Our project involved a combination of ML, computer vision, UI, and frontend development:

Model Training: We trained a YOLO11-based ML model on ASL alphabet data to recognize and interpret hand gestures.
Text Conversion: The recognized signs are converted into text, displayed to the user through the extension.
Extension UI: We created a mockup of our Chrome extension in Figma, then used HTML with CSS and Javascript to build a responsive interface into the browser.
Future Scalability: While the current version supports alphabet recognition, we’ve laid the foundation to expand into phrases, speech synthesis, and platform integrations like Zoom and MS Teams.

Challenges We Faced

SignScribe came with its fair share of hurdles.

Model Accuracy & Performance: Achieving consistent and accurate gesture recognition proved challenging, especially with the variability in hand positioning, lighting, and camera quality.
Chrome Extension Limitations: Chrome’s sandboxed environment limited our ability to access certain APIs, making it difficult to process and display results efficiently.
“ Real-Time Processing: Ensuring that video processing and ML inference happened in real time without significant lag was a major optimization challenge.
Time Constraints: Training and refining the ML model was time-consuming, making it difficult to implement additional features like adding ASL phrases before the hackathon deadline.

Looking Ahead

This is only the beginning for SignScribe. We plan to:

Expand beyond alphabet recognition to phrases and sentences.
Implement speech synthesis to give real-time voice to sign language.
Add native support for Zoom, MS Teams, and other platforms.
Incorporate multilingual translation, making Signscribe accessible worldwide.

We’re excited about the future of SignScribe and the impact it will have on making communication truly barrier-free.