Sense

Sense Homepage that comes up on opening
Login page for sense, assures user confidentiality, stores their past Sense visits to reduce redundancy.
Sense map of your local area to show the most affordable recommended healthcare facilities
Sense form to get more information about the user prior to their video answers
Sense chatbot assistant to give information to the user

What influenced you to create this project?

Our project, Sense, was created with a strong focus on making mental health care accessible to those who might otherwise be unable to receive it. Millions of people around the world face mental health challenges but lack the resources or access to traditional care due to barriers like stigma, cost, or geographic limitations. Many struggle without ever knowing what conditions they might have, leading to long-term consequences for their mental well-being. Our platform addresses this by offering a way for users to monitor and report their daily symptoms, providing early detection and helping them understand potential mental health concerns before they escalate.

What does your program do?

Sense is designed to empower users who may not have access to traditional mental health care by providing an affordable, accessible tool for early diagnosis. Using cameras, the platform analyzes subtle shifts in body language and facial expressions, combined with user-reported symptoms, to detect early indicators of mental health issues such as stress, anxiety, and depression. By leveraging advanced AI models, Sense gives users personalized assessments and recommendations, helping them understand their mental health status. It also provides suggestions for treatment options or connects them with affordable mental health resources, filling a critical gap for those who face challenges in accessing conventional care.

How did we build this project?

To achieve this, we built Sense using OpenCV and TensorFlow for real-time detection of facial expressions and body language, and integrated OpenAI and healthcare APIs for diagnosis and personalized recommendations. Our platform allows users to report daily symptoms, which are combined with the AI’s analysis to form a more complete picture of the user’s mental health. We also used the MapBox API to find nearby mental health resources and hospitals, and securely stored user data, including video submissions, using MongoDB and AWS S3. The front-end UI/UX was developed using React, TypeScript, TailwindCSS, Next.js, Node.js, and Mongoose to ensure an intuitive, user-friendly experience.

What were some challenges were faced?

One of the key challenges we faced was accurately detecting and analyzing emotional cues from facial expressions and body language using OpenCV and Python, as well as making sense of user-reported symptoms. Additionally, integrating the diagnostic algorithm from OpenAI and healthcare APIs in a way that provided reliable and actionable results was a complex task. Managing and securely storing user data, particularly videos, while ensuring that it worked seamlessly with other features of the app, was another significant hurdle. Finally, ensuring that all components—from video analysis to personalized recommendations—functioned cohesively in a user-friendly platform presented a significant challenge, but we are proud of how we overcame it.

What were our greatest accomplishments in this project?

Our greatest accomplishment is the creation of a platform that has the potential to make a real difference in people’s lives, especially for those who are unable to access traditional mental health services. We’re proud of how well our video analysis and diagnosis algorithms work together, offering users valuable insights into their mental health through subtle emotional cues and daily symptom tracking. The seamless integration of these functions into a cohesive, user-friendly platform was a major technical achievement. Most importantly, we are proud that Sense provides an accessible and affordable way for users to understand and manage their mental health.

What we learned

Through this project, we learned the importance of working with unstructured data, particularly in extracting meaningful information from videos and user-reported symptoms. Integrating multiple APIs and databases while ensuring compatibility across our front-end and back-end systems taught us the importance of smooth communication between complex systems. This project pushed us to think critically about how to deliver personalized care in a way that feels intuitive and helpful for users, and gave us valuable experience in creating an impactful health tech platform.

What's next for Sense

Looking ahead, our goal is to expand Sense’s ability to detect a broader range of mental health conditions by improving the accuracy and sensitivity of our vision and symptom analysis algorithms. We plan to further personalize the user experience by incorporating more in-depth questions and evaluations, enabling us to provide even more tailored recommendations. We also aim to make our platform available to a wider audience, focusing on individuals who may not have access to traditional mental health services, ensuring Sense continues to help those who would otherwise face their challenges alone.