Read the Room

Inspiration

One of the biggest problems with online learning is the lack of communication. During in-person classes, teachers can see their students are distracted and ask them to pay attention again right away. However, it's difficult to screen share, teach, use technology, and pay attention to how students are understanding at the same time. By the time the teacher recognizes that their students are distracted due to constant confusion, it is often too late. Our project aims to solve this problem.

What it does

It captures the student's face on a webcam and detects if the student is focused or distracted in real time.

How we built it

We built it using MATLAB on backend, and Unity on frontend. We trained a network using that differentiates a distracted face and a focused face using Squeeze Net on MATLAB, and applied it to a live face detection script that is provided by MathWorks. Then, we integrated this with our UI, with is built on Unity.

Challenges we ran into

It is the first time we worked extensively with MATLAB, and we did not have experience with deep learning prior to ths hackathon. We spent a lot of time reading documentations and asking mentor questions. We also have to search of datasets on the spot to train our model, which was no an easy feat.

Accomplishments that we're proud of

We are proud of how much we were able to learn and use within the 24 hours the hackathon lasted, as well as the final working model!

What we learned

We learned how to train a deep learning network on MATLAB, now to integrate a trained network into a live script, and how to integrate MATLAB with Unity.

What's next for Read the Room

Implementing facial expression recognition in addition to focused/distracted students, training a more robust network, having it integrate with software that are used in teaching institutions. (ie Zoom)

Built With

Submitted to

NewHacks - IEEE University of Toronto
- Winner Best Use of MATLAB
- Winner 1st Place Overall

Created by

I worked on back-end using MATLAB entirely. It was my first time using deep learning and detection algorithm, so I had to do a lot of research and learning on the post. In the end, I was able to train a neural network and use it in combination with a face tracking script that suits our project's needs.

Emily Chen
Helped research into deep learning and facial recognition. Learned about imbedding deep learning and adapting the code to our chosen datasets.

meneze73 Menezes
Created the front end in unity. Integrated the front end with the backend. Optimized the algorithm slightly.

Mark Halka
Researched image recognition and facial expression recognition. Found dataset necessary to run the deep learning algorithm and to train the neural network.

Alaa Hatoum