-
-
Group Pic
-
Architecture of Model
-
Metrics of Model
Inspiration
Did you know that over 70 million people around the world rely on sign language as their main form of communication? Despite representing approximately 1% of the world’s population, the deaf community still struggles with limited access to interpreters, stigmatization and bias, and isolation due to communication barriers. To us, a more interconnected world is one where all people are enabled to communicate and connect with others, regardless of any gaps in languages, cultures, and disabilities, and we want to make a difference.
What it does
For our project, we created an image recognition tool that is able to translate ASL signs into English letters. We believe our tool can be utilized in a variety of ways to improve awareness and education regarding ASL, such as being an educational tool for those interested in learning sign language.
How we built it
Initial ideation and planning were led by Raja and Milind in terms of the technology we wanted to build and furthering how we were going to accomplish such a task. Then to build this project, we split it into 2 main parts, a front-end GUI and a backend AI model. For the front end, Ajay and Steven used a simple python GUI library, tkinker, to make software to take an image input and output words from concatenating multiple outputs from the backend with the CNN model itself. Raja worked on the backend, which consisted of a 23-layer sequential model-based CNN in TensorFlow consisting of a 4-step convolution with max pooling and batch normalization into a 3-layer dense neural net that outputs 29 different classes associated with the different letters and symbols we have for ASL. Before training, we had to transform the data to a reduced dimensionality by resizing the images to 100x100 and converting them to greyscale, then 87000 images, 3000 images for each class, were used. This model has around 6 million trainable parameters and took roughly 2 hours to train with 6 epochs achieving a 97% accuracy overall.
Challenges we ran into
The main challenges we faced were testing and fitting a plethora of architectures to find an efficient structure for the CNN for training and accuracy, integrating the model after it was trained, and implementing a front-end GUI to complete the pipeline of input and output.
Accomplishments that we're proud of
We are proud of the 97% accurate CNN that we produced to classify letters in ASL from images into up to 29 different classes.
What we learned
We learned how to implement a CNN to solve real-life problems, and the limitations we have in terms of access to data and other technologies like making a live app.
What's next for ASLingo (AI-Powered ASL Translator)
From creating a mobile app to taking live feed video to translate ASL in real-time with autocorrection, there are a lot of ways to go about making this tool more viable. If we had more time, there are a lot of ideas we have for this project in the future beyond the initial steps we took. But this all would've definitely taken a lot of planning and time to be able to pull off, so we are definitely proud of the progress we made, but more could've clearly been done to make the program better.
Log in or sign up for Devpost to join the conversation.