In this day and age, those who suffer from speech or hearing impairment are not able to take complete advantage of online services such as YouTube. Currently, people belonging to this community are not able to effectively use platforms such as YouTube to produce content and convey their messages to a general audience. Similarly, those with the inability to hear have been shown to prefer reading in ASL rather than in English. We set out to bring a solution to this problem at hand by creating a web service that provides a way to translate between these two states. This is where Holly comes in. Given a video file of ASL gestures, Holly would return the video with audio and in the case when given an audio file, Holly would return a sequence of ASL images that can be used as captions on the actual audio or video file. Overall, we aim to bridge the gap and bring about equity to those who are affected by speech or hearing impairment, and give them a way to express themselves on the biggest platforms available out there. We believe that the complete implementation of our idea and project will help bring our communities together and eliminate any feelings of exclusion for those affected by hearing or speech impairment.
What it does
Holly is a web service that helps translate audio files into corresponding ASL images to be used as captions for videos or vice versa to add audio to video files containing ASL.
How we built it
Holly uses google's speech recognition API to convert audio files to text and uses the OpenCV library to stitch together images of these words in ASL. We used the Flask library to host our server and built an interface that handles the uploading of audio files and lets the user download the corresponding ASL image file.
Challenges I ran into
We were unable to complete the implementation of the ASL recognition system that would help convert sign language to audio. We also ran into issues while stitching together images for ASL from audio inputs, but were able to rectify this and build the system.
Accomplishments that I'm proud of
The team was able to pull together a large project in under 36 hours of work and we believe that this web service can really help establish social equity between those with hearing or speech impairment and those without. Having gone through the ups and downs of the project as a team, we are proud of what we were able to implement and will continue to improve on our ideas.
What I learned
I learnt that working as a team by balancing ideas, helping each other and using everyone's strengths and weaknesses is the key to success in any environment. 4 brains are better than 1, and bouncing ideas off each other and helping one another along the way helped us implement our ideas efficiently.
What's next for Holly
The next steps for Holly will be to completely implement the ASL to audio interpreter to fully accomplish our goals in mind and also be able to provide this as a service to be able to help bring sustainable social equity to our society.