Inspiration
I set down with what I believed to be the most socially beneficial problem statement to fulfill my aspirations of creating solutions that promote inclusiveness and eradicate biases to help those in need live a more fulfilling life.
What it does
Developed using Microsoft Azure's own AI service and flask framework as an initiative to aid individuals with hearing impairment for Code Without Barriers problem statements, PeaceSign facilitates the translation of speech into American sign language. The web application can transcribe and convert both short sentences and videos into hand signs after receiving speech input from the user.
How we built it
To derive hand signs as an output, the model first transcribes speech into text, and with the help of this article, I was able to reach greater heights with the project and enhance the capabilities of the model to receive longer speech inputs from 30min videos after some minor adjustments to the source code. For each alphabet in the transcription, the model fires an HTTP request to get the images from the classifier trained in customvision.ai before dumping the results as a downloadable image.
Challenges we ran into
Due to time limitations imposed by the hackathon and my lack of expertise in artificial intelligence software development, the original number of features including real-time speech-to-sign translation and sign-to-speech was reduced to only short speech and video translation. It was nonetheless, a very exciting and fruitful topic to conduct research on which can have produced good results if I had better time management.
Certain functions within this project take forever to load as an HTTP request is fired for each letter in the transcription, but I have managed to reduce the time taken from an hour to less than a minute. However, I still believe there are easier ways to speed up the application loading speed e.g. with class maybe.
The incompatibility of python modules within PeaceSign also posed a huge problem in the deployment of the solution to the cloud as Azure Web only offers Linux containers for python and after several unsuccessful attempts of deployment, I managed to pin down the issue by running through the dependencies and singling out the modules that were causing container failures before replacing them with alternatives that perform the same functions.
Accomplishments that we're proud of
Completing the task at hand for sure. Thanks to this hackathon, I was able to complete two certificate courses from Microsoft and grow my knowledge to achieve greater heights in my learning journey.
What we learned
The limitations and wonders of artificial intelligence, processes that can be simplified through machine learning, and services available on different platforms that can be utilized to streamline many existing functions.
What's next for PeaceSign
There are plans to migrate this project onto an android platform for extra portability in the future, though still in consideration as I am still in the process of learning and understanding Java language. If given the time and resources, I would definitely want to create a chatbot version of this project so it can easily be attached to any website for accessibility.
Log in or sign up for Devpost to join the conversation.