Inspiration

Lack of Inclusivity of the disable community realised through Market Research (online, offline). We met an audio-disabled person and understood the limitations and requirements of efficient communication with normal hearing persons.

What it does

The project concept translates Speech/Text to Sign Language and vice versa. The current project runs locally and we plan to integrate it with the Microsoft Teams App as well as a stand alone application promising digital accessibility across all platforms.

How we built it

We used Azure Cognitive services (Speech to text and vice versa) to convert output audio from a livestream to text, which is then sent to a pre-trained AI Model which is capable of converting Sign Language to Text and vice versa (under development)

Challenges we ran into

  1. No developer within the team.
  2. Dataset provided only labeled videos as Signing, Speaking and None. This would not help in involving the disabled community in communication.
  3. Too complicated and large model to complete with 48 hours.
  4. Lack of proper datasets. Our main goal to translate live conversations to ASL was not completely realised, as we couldnt train a proper model which could understand make sense of complete sentences in a conversation.
  5. Inaccessibility to the MS Teams API, thus not being able to integrate it with livestream videos.
  6. English and ASL not having a 1:1 overlap, thus we couldnt use datasets that were only converting single words to ASL.

Accomplishments that we're proud of

  1. Azure Cognitive Services code for Speech to Text/ Text to Speech
  2. Product Vision and Business Plan of open sourcing it.

What we learned

  1. How to plan and distribute work especially when team lacks AI/ML experience, and not lose motivation.
  2. To accept what the team lacks and leveraging what the team knows
  3. Inspite of advancements in AI and ML, top companies have still not solved the inclusivity problem.
  4. Business/Selling pitch is more important than a working prototype.

What's next for Online Accessibility for Deaf Community

  1. Digital Accessibility of disabled persons across all platforms (social channels) by strengthening the AI model and making the software patch.
Share this project:

Updates

Submission history