Inspiration
Everyone, especially adults, has been forced to work in a virtual setting as a result of the pandemic, using platforms such as Zoom, Skype, Webex, and other conferencing platforms. Those who are deaf or hard of hearing, unfortunately, face a greater challenge in communicating on these virtual platforms.
According to statistics, adults with hearing impairments cause a $12k decrease in household income on average. There are hearing aids for this, but 3.65 million hearing aids were thrown out in 2016 alone because many people do not want to wear them.
Based on the information we gathered, we decided to create a web application that would allow people with hearing impairments to communicate without difficulty.
What it does
Traverse is a video calling web application that converts ASL to speech and speech to text, allowing hearing-impaired people to communicate with others without difficulty.
Traverse boasts flawless communication thanks to the use of cutting-edge technology. Through the camera, Traverse will be able to see and read ASL hand gestures. Then, using machine learning and neural networks to process the data from the camera, our web application will attempt to recognise these hand gestures. Following that, the application will convert the AMS hand gestures to text, which will then be converted to speech.
How we built it
We used a python source code (Signum) that already had a basic ASL recognition scanner for the Speech to ASL and ASL to Speech conversions. We changed the code to remove the second and third guessing options. We made it a multi-user video chatting call with closed captions to add to the code. To convert the text into spoken words, we used Google's text-to-speech library (gTTS). Keras, a machine learning library that recognises the ASL alphabet and has a trained neural network with a database of ASL gesture images, was also used.
The website was built with React Native, and we used Figma to create a prototype and UI design for how we wanted the bridge to look.
Challenges we ran into
Beta testing was one of our most difficult tasks. We hoped to show and demo Traverse to people with hearing impairments and get feedback and advice on how we could improve our product and make it more accessible because our app's target market is primarily people with hearing impairments. However, due to the project's short timeline and COVID-19 concerns, we were unable to physically test and provide feedback to people with hearing impairments.
Another significant stumbling block for our team was the code. We found source code on Github that recognised ASL and deciphered the letter, but after running it, we discovered numerous errors that prevented the code from accurately identifying the ASL to letter. Our team had to devote a significant amount of time to first comprehending the code, then attempting to identify and fix bugs. To identify and fix the errors, our team relied heavily on online resources such as Stack Overflow, but the process took a long time.
Accomplishments that we're proud of
Coding a fully functional website to complement our idea is one of our major accomplishments for this project. We built this website with React Native, but most of our team members had no prior experience with it or knew how to code with it. Our team learned a lot about React, and we're proud of the website we built with it. We're also excited to try React Native in the future.
Another achievement was the Figma prototype we created. No one on our team had used Figma to create website UI designs before access, so we were all quite new to Figma and all of its features. We learned a lot about how to use Figma and create both informative and aesthetically pleasing UI designs along the way, and we were introduced to a new prototyping platform that we will definitely use in the future!
We're also proud of the fact that we were able to conduct a survey and receive 50 responses, the majority of which were in favour of the solution we're proposing.
What we learned
Throughout the process, our team was exposed to how many people face a variety of accessibility issues, and how these issues are frequently overlooked by large companies and corporations. The experience taught us that there is still a lot of work to be done in order to help people with disabilities and impairments access specific information and resources, and conducting extensive research on the subject has inspired our team to continue working on accessibility projects.
Our team also learned a lot about Figma and other prototyping platforms, as well as coding websites with React Native, thanks to this project.
What's next for Traverse
We want to keep working on Traverse in the coming weeks to make it more accurate and efficient at recognizing and understanding ASL. The code currently only recognizes letters, but we want to improve it so that it can recognize ASL for entire words, as this will be a much more useful feature. We also want to move away from making a website and instead make a browser extension so that Traverse is simple to use.
We also want to conduct a more in-depth market analysis, and we hope to speak with at least 50 people who have hearing impairments so that we can accurately determine what features we should include in Traverse.
Log in or sign up for Devpost to join the conversation.