SignSpeak: Bridging Gaps with AI/ML Technology

• 

  Translates the text into sign language

• 

  Converts Sign language into text

• 

  Learning the language for each text.

• 

• 

• 

  Sign Languages

Inspiration

The communication gap between deaf and hearing individuals has long been a barrier to inclusion, leading to social isolation and limited access to opportunities. Inspired by the real-world struggles faced by the hearing-impaired community and the lack of scalable, real-time, and user-friendly solutions, we set out to develop a platform that fosters inclusivity and breaks down communication barriers using AI and ML.

What it does

SignSpeak is an AI-powered communication platform that enables seamless interaction between deaf and hearing individuals. It includes:

Real-time sign language recognition using webcam input.

Speech-to-text conversion for live conversations.

Text-to-sign language translation using preloaded sign image datasets.

An intuitive interface for learning and using sign language in everyday scenarios.

How we built it

We used:

React.js for building the responsive and modular frontend.

Node.js for handling real-time backend processing and communication with ML models.

TensorFlow and Keras for training Convolutional Neural Networks (CNNs) to recognize sign language gestures.

OpenCV for real-time video feed processing.

Mozilla DeepSpeech for speech-to-text conversion.

Preprocessed ASL sign image datasets for text-to-sign translation.

Challenges we ran into

Ensuring real-time performance while maintaining accuracy under different lighting, speeds, and backgrounds.

Dealing with regional sign language variations and complex gestures.

Handling sentence-level translations where the image-based approach struggled.

Speech recognition challenges in noisy environments.

Accomplishments that we're proud of Achieved 85% accuracy in sign recognition and 12% WER in speech-to-text conversion.

Developed a lightweight system deployable on low-resource devices.

Created a user-friendly interface enabling effective two-way communication.

Received positive feedback from both deaf and hearing testers.

Built With

• keras
• languages:-javascript
• node.js
• opencv-speech-engine:-mozilla-deepspeech-dataset:-american-sign-language-(asl)-alphabet-tools:-github
• python-frameworks/libraries:-react.js
• tensorflow
• vs