Voice for the Deaf

Revolutionizing communication for the hearing impaired via an Apple Watch app that decodes sign language gestures to speech. The decoder uses a SVM to classify gestures and a LLM to create sentences.

Comment

Inspiration

Deaf and hard of hearing people rely on sign language to communicate. The use of sign language can be dated as far back as 5th century BC in Greece, and has undergone virtually no technological advancement.

Hearing loss is the most common neurological disorder, affecting an estimated 360 million adults and children worldwide, according to the World Health Organization. Yet there are no biological treatments to restore hearing once it's lost. Reference

Solution

With modern advancements in wrist worn wearables that can decode gestures from sensor data, large language models (LLMs), and a text to speech generator, we can make a real time sign language to speech generator to enable hard of hearing people to talk to anyone!

Our hand sign decoder leverages a SVM machine learning model to classify hand sign gestures from accelerometer and gyroscope data from the Apple Watch. It captures hand movements via the Apple Watch, translates them into word gestures, and then feeds the words into a Large Language Model (LLM) to capture the meaning of the stream of words in a sentence. A text to speech synthesizer vocalizes the translated sentence, ultimately providing a voice to those who struggle with traditional communication methods.

Wearable Data --> Gesture --> Word Classification

Wrist worn devices can record high density acceleration, gyroscope, and muscle movements! - The apple watch has built in APIs to read this data and classify gestures, and the technology is improving. - The myo arm band which was created prior to 2014 and later acquired by CNTRL-Labs, which was later acquired by Meta, is able to fully map a hand in 3D space - 96.16% accuracy for classifying 80 ASL signs from EMG (https://ieeexplore.ieee.org/abstract/document/7552525)

LLMs for Sentence Generation

LLMs (Large Language Models) have created a tremendous opportunity in natural language processing and syntactical language understanding. Any errors in gesture decoding will be corrected for by an LLM, which greatly improves translation accuracy.

How we built it

We developed a machine learning model trained on a vast dataset of hand sign languages, ensuring comprehensive and accurate recognition. The model was integrated with the Apple Watch's motion detection technology. Once the hand signs are identified and translated into text, the information is transferred to the LLM model for speech synthesis.

Challenges we ran into

  • Classifying finger positions from acceleration and gyroscope data proved to have very low accuracy. This pushed us to explore gestures which utilize arm movement more than finger movement. This issue can be mitigated by incorporating muscle sensors into the wrist wearable.
  • Time constraints forced us to pick a dictionary of just 5 words to classify, which meant our LLM would only be able to generate sentences with those 5 words. This limited the potential of the LLMs response, whereas in the real world, with a large dictionary, an LLM would prove to be more effective at sentence generation.

Accomplishments that we're proud of

We are extremely proud that we were able to incorporate sign language classification with an accuracy of 95% and a LLM that generates reasonable speech within just 2 days!

We were also impressed how easy it was to extend our sign language decoder from just 1-2 words to 5 words without a significant drop in classification accuracy!

We look forward to significantly improve the lives of deaf and speaking-impaired individuals!

What we learned

  • We learned that LLMs are largely unpredictable in their output generation, but can be tuned effectively with even ad-hoc prompt engineering.
  • We learned that it doesn't take a ton of data to effectively classify hand gestures!

What's next for Hand Sign Decoder

Looking ahead, we aim to refine our model to recognize even more nuanced hand signs and a larger set of words. We also plan to explore ways to expedite real-time translation and expand compatibility beyond the Apple Watch, to make our technology accessible to a broader audience.

Built With

Share this project:

Updates