SignALL

Inspiration

The inspiration for SignALL comes from the need to bridge communication gaps between hearing and deaf/hard-of-hearing communities. Millions of people worldwide use sign language as their primary form of communication, yet real-time translation tools remain limited. We wanted to create an accessible, user-friendly application that could instantly convert spoken English into visual sign language gestures, making conversations more inclusive and breaking down barriers in education, healthcare, customer service, and everyday interactions.

What it does

SignALL is a real-time speech-to-sign language converter that transforms spoken English into American Sign Language (ASL) gestures. The application:

Activates the device's microphone to capture live speech Transcribes the spoken words into text using speech recognition technology Maps each transcribed word to its corresponding ASL hand gesture using a pre-defined dictionary based on provided documentation Displays the appropriate sign language gesture images on screen in real-time, using AI image generator from the text description of the hand gestures in the provided documentation Provides a seamless, continuous translation experience as the speaker talks

This enables hearing individuals to communicate with deaf users, and helps those learning sign language to understand spoken conversations visually.

How we built it

SignALL was built using web technologies for maximum accessibility:

Frontend Framework: React for building the interactive user interface Speech Recognition: Web Speech API for converting speech to text in real-time Data Processing: Text extraction and image mapping to create a dictionary linking words to gesture images Image Display: Dynamic rendering system to show the appropriate sign language gestures Styling: Tailwind CSS for a clean, responsive design State Management: React hooks (useState, useEffect) to manage recording state, transcription, and gesture display

The core challenge was creating an efficient mapping system between words and gestures, which we accomplished by processing the provided documentation to extract word-gesture pairs.

Challenges we ran into

The live transcription was quite slow, and there was a huge database of hand gestures dictionary to loop through. Hence, runtime and time-lag proved to be a big issue in the application.

Accomplishments that we're proud of

To be able to bridge the gap between sign language users and verbal speakers.

What we learned

This project taught us valuable lessons about:

Accessibility Technology: Understanding the complexities of sign language and the importance of visual communication Speech Processing: Working with real-time audio input and the nuances of speech recognition APIs Data Structures: Designing efficient lookup systems for rapid word-to-gesture matching User Experience: Creating interfaces for assistive technology requires special attention to timing, clarity, and reliability Cross-disciplinary Development: Combining linguistics (sign language structure), computer science (speech processing), and design (visual presentation) Real-world Impact: How technology can meaningfully improve communication and inclusion for underserved communities

What's next for SignALL

Expanded Vocabulary: Adding thousands more signs to cover comprehensive conversational English Multiple Sign Languages: Support for BSL (British Sign Language), LSF (French Sign Language), and other international sign languages Animated Gestures: Moving beyond static images to animated demonstrations showing proper hand movements and transitions Sentence Structure Optimization: Implementing ASL grammar rules, which differ from English syntax Two-way Communication: Adding sign-to-speech translation so deaf users can respond Mobile Apps: Native iOS and Android applications for better performance and offline capability Machine Learning: Using AI to improve gesture recognition accuracy and handle regional sign variations Educational Mode: Interactive lessons and quizzes to help people learn sign language Customization: Allowing users to adjust gesture display speed and add personal vocabulary Community Contributions: Enabling users to submit new gestures and improve the database collaboratively

Built With

• css
• html
• json
• react
• shadcn-ui
• tailwind
• typescript
• vite