StudyVision

StudyVision utilizes ML and CV to analyze work habits, providing real-time insight into eye strain, focus, and mood. This data is especially useful for students with learning disabilities such as ADD.

Comment 3

Inspiration

StudyVision was inspired by the massive increase in remote learning caused by the COVID-19 pandemic. Fairfax County Public Schools, one of the largest school districts in the nation, the number of students earning F’s increased by 83% during remote learning, indicating a serious academic problem. Many students, especially those with learning disabilities such as ADHD, are finding it difficult to maintain focus during online learning while spending long periods of time staring at screens, which can increase eye strain and pose a threat to emotional health.

StudyVision was designed to help manage these issues by providing notifications and analytics regarding loss of focus, eye strain, and mood during study and classroom sessions.

What it does

StudyVision is a web application that provides three main features to users:

  • Focus and gaze detection - StudyVision detects a loss of focus when your gaze has shifted from the current task for an extended period of time and reminds you to remain focused and productive. It creates a heatmap showing where you’re commonly getting distracted as well as determining a measure of your attention/focus and graphing over time.
  • Eye strain - StudyVision monitors your blink rate, an effective tool to determine eye strain caused by extended screen time. Multiple studies and articles (1, 2, 3) show that visual fatigue caused by excessive blue light exposure can drop blink rates by about 60%. If the measured blink rate is lower than expected, StudyVision will notify the user to rest their eyes. This can help dramatically reduce visual fatigue, as many of the negative side effects can be prevented through occasional breaks from screen time.
  • Emotion recognition - Study Vision automatically monitors the different range of emotions you feel throughout your study session and notifies a parent or guardian during extended periods of high stress and negative emotions, while also providing you a detailed visual summary after your session.

After each study session, users can see a summary page of their study habits. This will include a chart of their focus over time, a heatmap of the locations that were looked at most often, a graph of eye strain over time, and a mood tracker. Users can also create accounts to keep track of past study sessions for future reference.

How we built it

  • Focus - Using algorithms for eye gaze tracking, we calculated the amount and direction to which the user is deviating from where they were previously focusing. We then aggregated these deviations into a heatmap to show the user’s tendencies. In addition, we used these deviations from the typical cluster where they were focused to create an overall focus metric so the user can see their focus levels in real-time along with their past data.
  • Eye Strain - We utilized a model to identify landmarks on the face, especially around the eye area. Using the six landmarks near the eye, we calculated the eye aspect ratio (EAR). The ratio is relatively constant when the user’s eyes are open, but will drop to zero when a blink occurs. Thus, we can use the ratio to find the number of frames in which the user is blinking, and then convert that into blink rate. The blink rate was then compared to the average blink rate of a person using a laptop, allowing us to label the user’s eyes as “strained” or “not strained”.
  • Emotion Recognition - We used an emotion recognition library that bundles a Keras model with a convolutional neural network to give a score to each of seven types of emotions at certain intervals. We aggregated these scores to give a percentage of each emotion experienced over the course of the study session. We then determined an optimal threshold of negative emotion scores required to send a notification to a trusted adult.
  • Website - We built the website using a Flask backend with a MongoDB database to store user data and session analytics. During a session, the front-end captured images which were encoded in base64 and sent to the flask server via. POST requests for analysis. The analysis was then rendered for display and figures were created for visually displaying data.

Challenges we ran into

  • Converting video and images to a base64 string so that our server can send them to our models
  • Getting the eye gaze tracker to be accurate
  • Compressing the application to allow it to run on the server
  • Integration of the web application and the features
  • Finding the optimal frame rate to give us enough information for all three features while not slowing down our application

Accomplishments that we're proud of

  • Successfully integrating all the components with the web app
  • Implementing three key CV features and successfully completing everything in our initial plan

What we learned

  • How to pass video and images through different parts of the application
  • How to use popular CV libraries for gaze detection, eye strain, and emotion recognition
  • Using Seaborn and Matplotlib to create heatmaps, graphs, and charts and sending them to the frontend

What's next for StudyVision

In the future, we hope to improve the reliability of our gaze detection tracker and the resolution of the blink detector to increase the accuracy of our results. We can also add more features to help productivity, including additional information specifically targeted to students with ADHD. Making user accounts more customizable with further analytics (such as study habit reports spanning multiple study sessions), would also help improve our product’s functionality.

Share this project:

Updates