NeuroDetect

Logo
Start Up Page
Disclaimer
Login Screen
Account Creation
Home Page
Patient List
Add a Patient
Patient Profile Part 1
Patient Profile Part 1
Results
Facial Analysis. Includes facial landmarks and type of gaze.

Inspiration

According to Alzheimer Society of Canada, by 2030, it’s projected that 955,900 Canadians will be living with dementia. Similarly, schizophrenia has significant social costs, such as on family members and caregivers, has an increased risk of involvement with the criminal justice system, and even has economic costs due to lost productivity, disability, and increased use of healthcare resources. One review estimates that the annual economic burden of schizophrenia in the United States is between $32.5 billion and $65 billion.

Mental well-being is as crucial as physical well-being, but there don’t yet exist any assistive screening mechanisms for them. Just like cancer screenings are recommended, we envision a tool that can serve as a preventative screening for the development of neurodegenerative diseases and mental health disorders.

What it does

NeuroDetect is an app that aids healthcare professionals get ahead of any potential health issues before they develop into something more serious. It utilizes two components to function accurately and minimize bias. Facial analysis AI is used to capture the changes facial expressions and gazes of patients who might be showing symptoms of dementia. The app identifies 36 distinct facial landmarks that are key to distinguishing how different mental health disorders can appear.

The other component of NeuroDetect identifies cognitive decline through audio recordings of neuropsychological tests. It uses an automated speech recognition software that helps mental health care professionals transcribe conversations of the patients. Then it detects patterns that assess the likelihood and severity of a patient’s cognitive impairment.

How we built it

The facial analysis AI uses OpenCV's pre-trained models to detect key landmarks on a person's face. These landmarks were identified through papers, which provided insight onto the most important aspects needed to determine various cognitive impairments. It then analyzes a person's gaze by using additional pre-trained models provided by OpenCV. This information is tracked over the duration of a video, and will be used to track progress and changes over time.

To analyze a patient's conversation data, we used a variety of APIs. We first converted the conversation audio into a text transcript through the use of Open AI's Whisper API. We then trained a machine learning model by classifying endpoints, and provided training examples in line with the cognitive impairments we wanted to focus on (dementia and schizophrenia). We then used Cohere’s Large Language Model (LLM) API , specifically their Sentimental Anaylsis tool, to anaylze a given text and conduct sentimental analysis on it to classify it based on our needs.

Challenges we ran into

When trying to use computer vision to analyze patient faces, we ended up trying and scrapping three different APIs before settling on a final one. We initially tried using Google Cloud, but this required credit card information and would charge after a certain number of API calls were used. After realizing this, we moved to Microsoft Azure, which didn't require a credit card and provided free credits for students. After taking the time to learn how to to use the Face service they provide and implement a basic use case to test, we ran into a 403 error. Research into this revealed that we need certain business permissions to use service, which we did not have and would not be able to get access to within the weekend. We tried using one additional API before finally settling on OpenCV's face detection models, which were both free and didn't require special permissions. Similar challenges were faced with the conversational analysis portion of our application. We initially attempted to use AssemblyAI in order to use sentiment analysis of the mp3 voice recoding. However, due to some API token authentication errors we were unable to resolve, we decided to switch to Cohere API and used its Sentimental Analysis tool for our project instead. While we ended up solving both these problems in the end, the process of finding an API that we could use took a huge portion of time that could have otherwise spent implementing and refining features.

Accomplishments that we're proud of

Within a short timespan, we managed to get our two main features working. We successfully managed to take in video input of a person and identify various facial landmarks, with a high accuracy level. We were also able to analyze a person's gaze based on their eye movements and direction. Considering this was one of our first times working with Artificial Intelligence, this was a huge accomplishment for us. Similarly, being able to analyze a patients conversation and categorize that into different emotions was a massive accomplishment not only due to the learning curve, but also because this was a key feature we needed in order to minimize bias in our application.

What we learned

This was one of our first times working with AI/ML models, so this was a steep learning curve. We learned how to use computer vision to analyze human faces, as well as in a more general sense. In addition, we became familiar with Cohere's API and how to train and use Sentiment Analysis models.

What's next for NeuroDetect

A key feature we want to implement is a post-appointment assessment, where we can summarize a patient's conversation with their therapist/psychiatrist and have them to answer a series of questions based on this, in order to test various abilities including memory. We also want to have a graph at the end of every appointment, which summarizes the facial landmark and gaze data which was analyzed. We will use this to track patient data over the span of multiple appointments, to see how their symptoms progress over time.