We were first inspired to build this application by the simple question, "How does a deaf person make a phone call?" From there we dedicated our time to researching American Sign Language, ASL, and how they communicate.
What it does
Wave Messenger is a basic messaging app with the unique feature of converting ASL to text, thus giving anyone the option to sign whenever they want.
How we built it
Wave Messenger is built with Android Studio and Tensorflow-lite. Together they form a powerful mobile app with infinite potential.
Challenges we ran into
Data: One of the reasons why this has not been done before is because there is no readily available data set for ASL, so one of our challenges was creating our own data set. However, this con turned into a pro because while other pre-trained models may be able to accomplish similar results, we have complete control over our data and thus can tweak and expand anyway we want in the future.
Android: Originally we planned on simply creating a standard keyboard that could be used anywhere on Android, but due to limitations in the operating system we could not access the camera in our custom keyboard.
ASL: Neither of us are experts in ASL, and in order to truly create the product we are hoping to create we would need knowledge of ASL that extends far further than just a vocabulary. For example, the form of ASL that would actually be used with Wave Messenger, and other forms of mobile ASL (ie. FaceTime), is one handed ASL and is actually a form of slang. There is no complete dictionary for one handed ASL.
Accomplishments that we're proud of
Our biggest accomplishment would be creating our own custom model with our own data set. One innovative technique we utilized was splicing videos for their frames rather than taking individual pictures; this allowed us to create a much larger data set than someone would expect within 36 hours.
What we learned
We learned so much about ASL, the different challenges that ASL speakers face, how its actually used, and so much more that we would have never known about had we not attempted this project.
What's next for Wave Messenger
Our main goal for the future is to find a new partner to act as our in-house ASL expert. This will allow us to implement all the features that we could not achieve before, simply because we did not know enough about ASL. On top of that we hope to continue pursing the idea of having a standard keyboard, or building out our messaging app to have functionality equal to mainstream messengers today.