Introduction
Although there are established techniques to translate between different languages, such as seq2seq models-- these technologies are only applicable to spoken language but not signed languages. We are working towards a vision where Deaf and hearing communities can communicate easier using modern technologies. Such technology would be almost synonymous with Google Translate but for signed languages. We will be implementing it by creating a model that can translate between images of signed letters/words to their respective written counterparts. This would be a classification problem, aiming to classify all letters of the Arabic sign language in our case into their written representations.
Challenges
So, far one of the challenges that we have faced is determining the right preprocessing techniques and determining the best filtration technique to apply to images to help the model get the best results. Moreover, we have had difficulty pickling the data. Our pickled file is greater than the size of the actual images. Lastly, we also had trouble aligning all the images to a standard array size because some images had slightly different sizes than the standard 64x64 pixel value. We also experienced that some data was corrupted and could not be resized to the consistent 64*64 pixel values, and we finally ended up skipping that to maintain consistency across the data that the model is fed.
Insights
We currently have no concrete results to show. It is our next step in the project to implement CNN and test the results. Our team is currently working on it, and we expect to have some results to show by the end of week before we meet up with our TA for the second checkpoint. Something not so concrete yet promising is that after plotting our filtrated images, we can see that our edge detection is working properly, we still have to see if that would affect our performance at all in the future.
Plans
We are on track for the project. So far, we have been following the plan as expected, and we have not made any major changes along the way. We expect to continue on track and to be able to implement our first iteration of the CNN structure by the end of week. We expect that we would need to allocate more time for testing the different variations of the CNN model that we come up with on different filtration techniques on images. This is what we expect to consume much of our development time for this project.
Log in or sign up for Devpost to join the conversation.