Attention Tester

Upload a video of your student looking in the camera and wait for the magic to happen!

Inspiration

With remote classes due to COVID-19, teachers are faced with unprecedented challenges. One of which is to ensure that their students stay engaged and concentrated throughout the class. To do so, they often have to switch between tabs to check on all their students’ camera, which can be quite tedious.

What it Does

We want to ensure a smooth transition between in-person teaching to online learning for both students and teachers. Hence, we created "Attention Tester", which provides feedback to teachers concerning their students' attention span throughout a course. First, the user (teacher) uploads a video of his/her student (all EXT supported). Our AI model then analyses the student's attention span throughout the video. It then redirects the user to the results page. This page displays the final score and a progression bar that shows the student’s average attention span throughout the video. Additionally, a graph shows the student’s overall attention span over the time of the video. Finally, a snapshot timelines provides real-time results

How We Built it

It uses MTCNN,a convolutional neural networks to detect faces as well as their landmarks such as nose, eyes, and lips. First, we take into consideration of the person looking away from the camera by using a "nose face score", which is the result of normalizing the distance between the nose and the face centroid. Then, we have a "face frame score" that calculates the normalized distance from the person's face centroid to the frame centroid. In other words, it detects how aligned a person is with respect to the centre of the camera. Finally, we take the minimum of these two scores and it becomes the attention score, which is given as a feedback for the end user. This feedback will help teachers adjust the pace of their lecture and offer them some peace of mind.

Challenges We Ran Into

We had to decide between a trade-off between speed of the calculation and its accuracy. We have decided to prioritize the accuracy finding metrics that worked in estimating user attention (gaze estimation for instance did not work) It is not as stable as the attention score given by the head moving sideways (nose, head moving away from frame). Another challenge is finding the optimal AI model (cascade vs MTCNN)

Accomplishments that I'm proud of

We managed to produce a finished product in time. We also managed to find the optimal AI model to detect faces (MTCNN) and design a user-friendly website.

What We Learned

First, we learned how to use MTCNN to detect nose position as well as the face bounding box. We also learned how to use head pose estimation with OpenCV by using PnP as well as gaze prediction by splitting the eye into a grid of 3x3 and averaging the pixels brightness. Also, we learned that simple solutions are often the most stable one and the one that works the best. We also learned how to handle file uploads in the frontend in React as well as in the backend in Flask.