Inspiration
Lack of Inclusivity of the disable community realised through Market Research (online, offline). We met an audio-disabled person and understood the limitations and requirements of efficient communication with normal hearing persons.
What it does
The project concept translates Speech/Text to Sign Language and vice versa. The current project runs locally and we plan to integrate it with the Microsoft Teams App as well as a stand alone application promising digital accessibility across all platforms.
How we built it
We used Azure Cognitive services (Speech to text and vice versa) to convert output audio from a livestream to text, which is then sent to a pre-trained AI Model which is capable of converting Sign Language to Text and vice versa (under development)
Challenges we ran into
- No developer within the team.
- Dataset provided only labeled videos as Signing, Speaking and None. This would not help in involving the disabled community in communication.
- Too complicated and large model to complete with 48 hours.
- Lack of proper datasets. Our main goal to translate live conversations to ASL was not completely realised, as we couldnt train a proper model which could understand make sense of complete sentences in a conversation.
- Inaccessibility to the MS Teams API, thus not being able to integrate it with livestream videos.
- English and ASL not having a 1:1 overlap, thus we couldnt use datasets that were only converting single words to ASL.
Accomplishments that we're proud of
- Azure Cognitive Services code for Speech to Text/ Text to Speech
- Product Vision and Business Plan of open sourcing it.
What we learned
- How to plan and distribute work especially when team lacks AI/ML experience, and not lose motivation.
- To accept what the team lacks and leveraging what the team knows
- Inspite of advancements in AI and ML, top companies have still not solved the inclusivity problem.
- Business/Selling pitch is more important than a working prototype.
What's next for Online Accessibility for Deaf Community
- Digital Accessibility of disabled persons across all platforms (social channels) by strengthening the AI model and making the software patch.
Built With
- azure
- c#
- machine-learning
- mediapipe
- opencv
- python