Not everyone has the right state of mind and body in this pandemic but almost everyone has a webcam.
In a digital world affected by COVID-19, digital diagnosis and fitness both mental and physical is more necessary than ever. The pandemic has caused an influx in hospital cases, and the limitations on hospital beds have people wondering whether their symptoms are severe enough to warrant a doctor's appointment. Meanwhile, others experience ailments but are unable to afford a visit to the doctor due to a lack of or poor health care. We are in alarming need of free and practical digital healthcare for keeping in touch with our medical needs. More than 2 in 5 people globally reported symptoms of depression and anxiety during the pandemic. As students, we have realized that a lot of our time is spent in front of a laptop and first-hand watched our mental health deteriorate during the pandemic. Even That’s why having an application like Medisure is so important!
What it does
Medisure is an app that allows users to regularly check up on their physical and mental well-being without needing to visit a doctor unless needed and enjoy AR/VR fitness demos.
We analyze your health using just two inputs - your video and your text. The features of our product are numbered below:
First, it can register the "mood" of a person. This is very important to provide automated action and suggestions from the website and to the user. The product can detect a total of 4 moods: happy, sad, angry, and neutral through the webcam of the user. With different moods, the website can perform different actions. If the user is happy, the website suggests if it should play the user's favorite Spotify playlist. If the user is sad, the website suggests if it should tell the user a joke and play upbeat music. It can also ask "check-up" questions. The website also plots a graph of your emotions in a fixed time necessary for regular mental checkup and even suggests to take counsellor help by booking a call.
There is a friendly chatbot to answer your questions and can even alert the user's relative/friend through SMS if the user gets suicidal and depressed to a certain level. This can be of great help to save someone's precious life.
We have a symptom checker section where you can just give your inputs step-by-step according to your issues and complaints to get a probable disease diagnosis and department of medicine to be contacted in the case of severity.
We have a flowing UI catering to the needs of the users. We have tried to make the overall look and feel to make it calming, soothing, safe and engaging enough for our users.
We also have a AR/VR fitness workout sections.
How we built it
Video Emotion Detection
Emotion detection was done using Face recognition API. We used a custom open-source model that we found online to help us process user's emotions. We then extracted the user emotions from the API and connected them to timers on the frontend. This allowed us to have a Hashtable (using chart.js) that contains all of the data on how many minutes the user feels each emotion during the duration of the video recording.
We suggest playlists according to the mood of the user using Spotify API.
Chatbot alert system
Symptom Checker System
AR/VR Fitness Zone
We used echoAR to implement few workout demos in AR/VR so that our users can learn exercises and stay fit from the ease of their houses.
Challenges we ran into
We were literally racing against time. The building of a good UI with all kind of functioning of different APIs and integrating each of them according to our user's need was indeed a big challenge for us. This was our first time utilizing Firebase to implement the login authentication of our website. We took the time to learn and understand how Firebase works and tried implementing it into our website. There were a variety of bugs to fix till the last moment. Unfortunately we were unable to deploy our web app successfully due to the problems in using multiple pages and PHP files. Implementing AR/VR through echoAR was also very difficult due to various errors showing up.
Accomplishments that we're proud of
This project was especially an achievement for us because we created many additional components on the front-end in order to have a completed project. We are also particularly proud that we were able to include a vast array of different features for the user, which we felt makes Medisure such a serviceable form of digital healthcare. Lastly, we think the impact our project could have is a significant accomplishment. Especially in the digital age combined with remote day-to-day communications, this could really be a product that people find useful!
What we learned
We have a learnt a plenty much investing time and efforts in this project. Use of various APIs and integrating them plus a UI design to implement in front-end in such a short span of time was unbelievably challenging.
What's next for Medisure
Medisure has so many useful features, so we'd love to further expand the project to handle direct 1:1 video calls with certified doctors and counsellors if required! Overall, we hope that one day this project can be widely used as a practical and accurate resource for body and mental checkup.