Understand student engagement with teaching materials through heatmap analytics:
Identifying distracted students during online classes through computer vision algorithm
Improving lesson plan with end-to-end student engagement insights

Nirvana - Reimagining classroom engagement for teachers

Inspiration

Due to the pandemic, many school systems around the world have been upended, resulting in teachers having to adopt remote and hybrid learning fast. According to The Harvard Gazette, more than 90% of the world’s schools closed at the height of the pandemic, and about 213 millions students are still fully remote. With the uncertainty of the future scenarios of the pandemic, and with remote learning here to stay, teachers are relying more on alternative approaches such as the flipped classroom approach. While this learning format has been effective, teachers are still finding it incredibly challenging to ensure their students are engaged, especially with the lack of face-to-face lessons.

A user interview with one of our group member’s parents - who are both primary school teachers, showed that one of most significant challenges for teachers is maintaining their student’s attention, engagement, and distractibility levels in the flipped classroom. “Being unable to see our students face-to-face has made it more difficult to understand their attention and engagement during lessons. This makes the planning of teaching materials, and knowing if our students are paying attention very challenging. The problem is especially challenging when I have to look out for over 30 students at a time,” shares Jenny.

What it does

Frontend Demo

Nirvana allows teachers to understand the attention and engagement levels of their students during flipped classrooms. Our solution identifies when students are distracted during their own learning of class materials, and during online class time. Through detecting student’s eye movements and yawns using computer vision algorithms, and through identifying interactions with content through heatmap analytics, Nirvana allows teachers to understand students’ engagement levels throughout the entire flipped classroom cycle.

Understand student engagement with teaching materials through heatmap analytics: During the self-directed learning portion, teachers will be able to see how the students interact with class materials, through understanding their clicks, taps, scrolling behaviors, or average drop-off rate, to understand what content their students are interested in. Nirvana gives teachers an aggregate of engagement across their teaching materials, allowing teachers to identify areas of weakness or opportunities in their teaching materials. This can allow them to better plan and design their teaching materials to increase student engagement.

Identifying distracted students during online classes through computer vision algorithms: During online classes, teachers will receive pings that remind them to check in with their students when our computer vision algorithm detects that students are distracted. At the end of the class, teachers also receive a custom report that shows a summary of their class's attention and engagement during the class. Metrics such as distraction rate, average attention span, and engagement level will be provided along with key insights and visualization charts that can help teachers improve future classes

Improving lesson plan with end-to-end student engagement insights: Tracking the engagement levels from all touchpoints of the flipped classroom will enable teachers to have a good overview of their student’s engagement; from the teaching materials to the online classes. Having key metrics and visualisation of data through the Nirvana dashboard or custom report enables teachers to easily access how engaged students are in their teaching materials and online classes. This will allow the teachers to identify improvement areas in their current lessons, utilise data to plan for future lessons, and also track their student’s engagement progress over time.

Protecting student’s data with Incognito activation: This allows teachers to collect anonymised user behaviour to get an overview of the student’s engagement, while protecting end-user privacy. With a private and protected version, it makes sures that Nirvana is compliant with the different privacy laws in different countries.

Benefits of Nirvana

Increased engagement levels, and better student-teacher relationship: Having insights into student engagement levels from the teaching materials to online classes allows teachers to understand more about their student’s individual needs and requirements. This allows teachers to develop more personalised lessons, and also enables them to reach out to students who might require more help.

Reduced time and cost in lesson planning: Teachers will be able to save time and cost in lesson planning when they use the insights to only focus on what is necessary in the teaching materials and lessons.

Nirvana’s data usage

Data collection: The raw data collected is primarily video data. The video data is analyzed and by detecting student eye movements and yawns using computer vision algorithms, and identifying interactions with the content using heat map analysis, Nirvana can understand student engagement throughout the Flipped Classroom cycle.

Data privacy and storage: To ensure data privacy, the data collected will be stored on a secure cloud server to ensure data privacy. In addition, only class insights will be stored, and raw video data will be removed. Teachers will also only be able to access the data of students they are responsible for. In addition, as our solution uses facial images, and other biometric information, we will ensure that Nirvana complies with the local General Data Protection Regulation (GDPR). We will also look to gain the required approval from the authorities, and parents, and also conduct sufficient impact assessment to ensure that Nirvana meets the GDPR requirements.

How we built it 🛠

For the front end, most of the prototype was built on Figma. We built a frontend on Angular and deployed it on firebase. *Frontend demo can be viewed here : * https://nirvana-72ef2.web.app/

We built the engagement tracking software with computer vision (OpenCV, Tensorflow(backend) and keras.

This following is how we approached develop Nirvana:

Step 1: We Used OpenCV Haar Cascade Face Detector to detect an individual's face.

Step 2: After detecting the face, OpenCV then locates the individual's eyes. A pre-trained Convolutional Neural Network (CNN) is used to predict whether a person is distracted using a binary classifier. The Default CNN was trained on Asian’s faces, in which we took both our own faces, and open data sets of people. Faces trained either looked distracted or focused during online meetings.

Click here to view demo of the our attention tracking software

Computer Vision

Challenges we ran into

Product-fit challenges: adapting solution to keep up to date with latest learning formats: Initially, we built a solution that can only be used for online classes. However, after speaking to teachers, we quickly learnt that the formats of learning can be diverse depending on the schools, lesson type or teacher. In addition, teaching formats are fast-evolving, and changes based on the needs of the students. While we have adapted our solution to be used in a flipped classroom format, we understand that it is necessary that our solution is able to remain adaptable and integratable to the changing teaching formats in the classroom.

Technical challenges: having limited training data for our computer vision algorithm: As we are only training the online dataset as well as our own faces, the accuracy rate reflected by our algorithm might change if exposed to faces or different nationalities or age. However, we are confident that with more training, we will be able to ensure our algorithm is able to obtain a high accuracy rate with faces of different nationalities or age.

Implementation challenges: Ensuring compliance with GDPR: Ensuring that Nirvana complies with the local General Data Protection Regulation (GDPR) could be a potential challenge we could run into during the implementation of our solution. As our solution uses facial images, and other biometric information, we will need to ensure that we gain the required approval from the authorities, and parents, and also conduct sufficient impact assessment to ensure Nirvana meets the GDPR requirements.

Accomplishments that we're proud of

Validated solution through user testing: The initial mockup of Nirvana was used to solicit feedback from our target users - teachers. With their feedback, we were able to better tailor our solution to the real-life issues the teachers face when engaging with students in flipped classroom environments.

What we learned

Importance of a holistic view in solution development: Having a team made up of individuals of different backgrounds (design, software development, and business background) allowed us to bounce ideas off each other to come up with a solution that considers various perspectives. We saw the importance of utilizing different technical methods such as computer vision, and heat map analysis to collect important data points, significance of having a good design that user-friendly, and also the criticalness of ensuring that our solution abides with the relevant compliance and privacy laws to ensure it is feasible to be used in the real-life world.

What's next for Nirvana - Reimagining classroom engagement for teachers

Pilot-testing in tuition centres: We are confident of the impact Nirvana will have on improving student’s engagement levels. Testing out Nirvana on a small scale will test out our prototype, and iterate to make improvements so that our solution can be commercially launched and used in schools throughout Singapore.

Other use cases such as mental well-being detection: Students are experiencing increasing stress levels, and lower engagement with learning since the pandemic, according to Stanford. Locally, more teens are seeking help at the Institute of Mental Health (IMH) for school-related stress, according to The Straits Times. With this rising trend in a decreased mental-being, Nirvana can also be adapted to help teachers identify if their students are displaying signs of mental distress. Being able to detect signs of mental distress can allow teachers and parents to help their students and children cope, or seek the required treatment if necessary.