During this unusual pandemic due to COVID-19, all of us around the world were under Quarantine. it can be much harder for those with disabilities or those who need accommodation in reading to perform well academically. At home, many students had to go through a harder learning process due to the lack of technical and educational support. Therefore, we we came up with Edu Audio, which is a Web/Mobile application that allows students listen to both what the professor speaking/writing through Zoom.
What it does
Our Software provides an interface for the professor to take notes. A professor can start a session (which will generate a join code) and as the professor writes, he can click on the convert button which will convert what the professor has written on the app to plain text, through a convolutional neural network. The professor can then pass on what he had written on the board to the DSP students by clicking on "send to student".
A student who is having a hard time seeing the screen can now reach into his/her app or computer and listen to the notes that the professor has written. After the lecture, the professor can close the session which will save a copy of the notes in our firebase database, which both the student and the professor can access with the app.
We put extra emphasis on the UI of the web/mobile app by coding in speech directions. We made big buttons and clicking on a button will tell the user what that button does.
How I built it
For our web app, we used basic web frameworks for the front end and flask for the back end. For our IOS app, we used React-Native that allowed us to utilize the cross-platform language for integrating the application in both Android and IOS. Firebase live database was used to tie the mobile and web app together. Our neural network was originally trained on computer text by a library called pytesseract. We had to retrain it using MNIST datasets so it could recognize handwritten images better.
Challenges I ran into
For more than half of us, this is our first hackathon. Also, we used lots of new technologies. Our teammates did not have any experience in neural network before, which forced us to watch a lot of tutorials. Furthermore, integrating the web-application with that of the mobile had to go through various difficulties due to the use of Firebase and frequent data updates.
Accomplishments that I'm proud of
This was a very ambitious project and we are proud that we pulled it off. Even though it isn't practically usable (if you compare it with something like notability), all the basic functionalities are there are we hope we can work on this application in the future to make the features more user friendly.
What I learned
We learned web/ios frameworks, integration between these devices, and some basics in unsupervised learning.
What's next for Edu Audio
We really need to get a better gpu, add more layers to our network, and retrain the model using only handwritten langauges for a lower loss rate.