COVID-19 has drastically transformed education from in-person to online. While being more accessible, e-learning imposes challenges in terms of attention for both educators and students. Attention is key to any learning experience, and it could normally be assessed approximately by the instructor from the physical feedback of students. However, it is not feasible for instructors to assess the attention levels of students in a remote environment. Therefore, we aim to build a web app that could assess attention based on eye-tracking, body-gesture, and facial expression using the Microsoft Azure Face API.
What it does
C.L.A.A.S takes the video recordings of students watching lectures (with explicit consent and ethics approval) and process them using Microsoft Azure Face API. Three features including eye-tracking, body posture, and facial expression with sub-metrics will be extracted from the output of the API and analyzed to determine the attention level of the student during specific periods of time. An attention average score will be assigned to each learner at different time intervals based on the evaluation of these three features, and the class attention average score will be calculated and displayed across time on our web app. The results would better inform instructors on sections of the lecture that gain attraction and lose attention in order for more innovative and engaging curriculum design.
How we built it
- The front end of the web app is developed using Python and the Microsoft Azure Face API. Video streaming decomposes the video into individual frames from which key features are extracted using the Microsoft Azure Face API.
- The back end of the web app is also written with Python. With literature review, we created an algorithm which assesses attention based on three metrics (blink frequency, head position, leaning) from two of the above-mentioned features (eye-tracking and body gesture). Finally, we output the attention scores averaged across all students with respect to time on our web app.
Challenges we ran into
- Lack of online datasets and limitation on time prevents us from collecting our own data or using machine learning models to classify attention.
- Insufficient literature to provide quantitative measure for the criteria of each metric.
- Decomposing a video into frames of image on a web app.
- Lag during data collection.
Accomplishments that we're proud of
- Relevance of the project for education
- Successfully extracting features from video data using the Microsoft Azure Face API
- Web design
What we learned
- Utilizing the Face API to obtain different facial data
- Computer vision features that could be used to classify attention
What's next for C.L.A.A.S.
- Machine learning model after collection of accurate and labelled baseline data from a larger sample size.
- Address the subjectiveness of the classification algorithm by considering more scenarios and doing more lit review
- Test the validity of the algorithm with more students
- Improve web design, functionalities
- Address limitations of the program from UX standpoint, such as lower resolution camera, position of their webcam relative to their face