universal sign language assistant

Inspiration

The idea behind the project is that users who are hard of hearing should be able to access an on-device sign language translator for WhatsApp, YouTube, or any other app that provides accessibility tools like closed captions.

What It Does

The app:

• Intercepts captions from apps like YouTube
  
• Generates tokens by breaking captions into words or phrases
  
• Uses a mapper to map tokens to gestures
  
• Generates an animation based on the mapped gestures
  

How We Built It

• Built using Jetpack Compose
• Exposes an Android Accessibility Service to intercept captions
• The service broadcasts captions to a mapper
• The mapper tokenizes captions, maps gestures, and animates them

Challenges We Ran Into

• Intercepting captions: Android documentation was unclear
• Designing the mapper:
  • Faced the classic producer overwhelming the consumer problem
  • Had to explore backpressure, rate limiting, and queues

Accomplishments That We're Proud Of

• Caption interceptor for YouTube
• Tokenizer
• Animator (though it still needs improvement)
• Moving from idea to a working prototype

What We Learned

• Android Accessibility Services
• Android app overlays
• Backend concepts such as:
  • Backpressure
  • Rate limiting
  • Queues

What's Next for Universal Sign Language Assistant

• Improve the animator
• Enhance the tokenizer using machine learning
• Add support for African languages
• Allow users to add new gestures

Built With

• android
• jetpack-compse
• kotlin