One of the biggest problems with online learning is the lack of communication. During in-person classes, teachers can see their students are distracted and ask them to pay attention again right away. However, it's difficult to screen share, teach, use technology, and pay attention to how students are understanding at the same time. By the time the teacher recognizes that their students are distracted due to constant confusion, it is often too late. Our project aims to solve this problem.
What it does
It captures the student's face on a webcam and detects if the student is focused or distracted in real time.
How we built it
We built it using MATLAB on backend, and Unity on frontend. We trained a network using that differentiates a distracted face and a focused face using Squeeze Net on MATLAB, and applied it to a live face detection script that is provided by MathWorks. Then, we integrated this with our UI, with is built on Unity.
Challenges we ran into
It is the first time we worked extensively with MATLAB, and we did not have experience with deep learning prior to ths hackathon. We spent a lot of time reading documentations and asking mentor questions. We also have to search of datasets on the spot to train our model, which was no an easy feat.
Accomplishments that we're proud of
We are proud of how much we were able to learn and use within the 24 hours the hackathon lasted, as well as the final working model!
What we learned
We learned how to train a deep learning network on MATLAB, now to integrate a trained network into a live script, and how to integrate MATLAB with Unity.
What's next for Read the Room
Implementing facial expression recognition in addition to focused/distracted students, training a more robust network, having it integrate with software that are used in teaching institutions. (ie Zoom)