-
Lip tracker
Inspiration
We were thinking of how people with hearing disabilities must use speech to text which can be a very unreliable method because of the ambient noise. So we thought of optimizing this method by tracking the person's lips and converting this to text.
What it does
This project tracks the speaker's lip movements and converts it to text for the user to see.
How we built it
The first step was writing a program which tracks the user's lips using the face_recognition module and Opencv. Once we were able to successfully do that we started compiling training data for our neural network and then we started training it. After experimenting with different epochs, learning rates, and decays we decided on an optimum level for all of them with one we were able to achieve maximum accuracy.
Challenges we ran into
The main challenge was time, compiling training data and writing the neural network along with tracking the lips was very difficult. Another major problem we ran into was installing the packages and the versions of python because some of the packages were not compatible with the versions of python.
Accomplishments that we're proud of
Making the 10,000 training samples ourselves and writing the neural network in 24 hours where we encountered multiple problems was something that we are quite proud of.
What we learned
In our team, we had amateur coders and they learned the basics of neural networks and we learned the massive problems associated with the installation of packages and the how annoying the different versions of python can be.
What's next for Lip 2 Text
Improving the neural net's accuracy and possibly having a much better UI.
Team members
Aditya Chaudhary: 573 Yashas Jain: 571 Siddharth Srivastava: 671 Aryan Mohan: 630
Categories
We believe that this hack is quite socially useful thus we are competing in that category and in the pre-university category
Log in or sign up for Devpost to join the conversation.