Inspiration

In Canada, 17.2% of the country's population were aged 65 and older, and this number is only set to increase in the next few decades. Individuals over the age of 65 have a higher risk of developing medical conditions such as coronary artery disease, arthritis, diabetes, obesity, and neurological diseases. 85% of the aged Canadian population suffers from at least one chronic condition. Seniors suffering from chronic diseases require regular health check-ups every 3-4 months for proper management of medications, vital signs, and lab values. These conditions impact the quality of life and ability for seniors to perform in everyday activities, thus increasing the need for caregivers to assist with daily routines on top of managing complex medical conditions. Furthermore, evidence suggests that caregivers enter their roles with little support and therefore carry high rates of mental and emotional health problems as a result.

With the onset of COVID-19, the elderly's ability to access usual medical care and emotional support has drastically decreased, and the communication with caregivers and family is impaired. Seniors are advised to stay at home and may feel isolated from their family and friends, leading to struggles with mental health on top of existing feelings of hopelessness and possible grief from the loss of loved ones. Moreover, as a result of impaired health care, the elderly are also more susceptible to exacerbating chronic conditions that raise concerns for future life-threatening complications from the coronavirus. Caregivers too may experience increased stress due to barriers in remote support for the care recipients health and daily needs.

These problems all point to the need for a solution in developing communication channels between elderly and caregivers for health management, while making remote support and health care accessible. Mobile health (mHealth) interventions using smartphones have proven effective for monitoring mood and health symptoms, while also providing a platform for communication and support for mental health concerns. However, these applications are not always accessible for the elderly population. Finger sensitivity and mobility can be an obstacle for the elderly as it impairs their ability to interact with apps. Features such as larger font size, high contrast, and text to speech functionality are often neglected due to the lieu of modern design trends intended to appeal to younger audiences.

Therefore, we designed our app, miia (Medical Intelligently Applied) to be accessible and usable by most seniors, while providing a platform to aid in remote caregiving. Miia is an application that will help track and manage both physical and mental health conditions for the elderly population. For instance, we implemented a Chatbot function to ask seniors about mood and emotions, while also providing a means to input health measurements such as vital signs. The chatbot can be made to speak aloud, while the senior can utilize their voice which is then converted to text. Furthermore, our app has an additional function to track mobility and activity functions of our users through drawing data from the built-in accelerometer, gyroscope, and other smartphone sensors. This will help us predict activity, encourage exercise, and potentially prevent frailty and traumatic falls with seniors.

What it does

The system leverages AI technology to analyze data collected from facial recognition, speech recognition, wearable devices and IoT on a daily basis, and alert the caregivers if there are any identified risks. The platform also provides a way to facilitate communication between caregivers and care recipients, while aiding with health management to alleviate caregiver stress.

Our current figma prototype serves as a better representation of the app's final functionality and design. In contrast to the web application the prototype is developed for mobile devices to better serve the elderly through prioritizing convenience and mobility. The prototype itself is fully interactive as users have the ability to click, scroll and drag through both caregiver and patient interfaces.

App Guide

Caregiver

  1. Signs up in the app and makes a profile for both themselves and their care recipient.
  2. After choosing the caregiver option, they will set up an account with their email and phone number, and set a password.
  3. Then, the caregiver will add the patient’s name and phone number.
  4. They can then add the pre-existing medical conditions of their care recipient. In this case, the preset conditions are common chronic diseases but there is also the option to add more conditions and background information.
  5. The caregiver can choose important metrics to monitor for certain chronic conditions, such as blood sugar level for diabetes, or mood for depression.
  6. After adding the background information for the patient, a unique pin will be generated for connecting the caregiver with the care recipient.
  7. A confirmation screen will also show the patient’s conditions and metrics to follow. If there are multiple care recipients, the caregiver can add another patient.
  8. On a daily basis, caregivers log in and monitor health of care recipients, the most important metric on display. The red notification symbol indicates a warning that requires caregivers to follow up on a metric.
  9. In the patient profile, the caregiver can change or add more metrics to monitor, chat with the patients, or edit the patient profiles.

Elderly/ Care Recipient

  1. Care recipient received a text message from the caregiver with his/ her unique pin. If a senior is unfamiliar with technology, the caregiver can help him/ her to set up the app.
  2. Choose to sign up as a patient, and enter the pin received.
  3. Our chatbot, guide seniors through the whole health checkup process on a daily basis
  4. The patient can choose to text or speak to the chatbot.
  5. Miia will proceed to initiate the process of health check by taking their facial image
  6. Miia will first ask a few questions regarding their physical and mental health, such as body temperature, blood pressure, or mood and the senior can input manually or tell miia their measurements. For voice inputs, Miia will repeat the measurement to verify.
  7. Depending on the needs of the senior and caregiver, the chatbot can also ask about other metrics, give reminders, or chat with the senior.
  8. After health check, users will be redirected to a health overview which summarizes results for the senior.
  9. Key metrics of seniors are shown in measurements. If the user is interested in knowing more of a particular metric, they can click the metric and look into the details.
  10. If seniors have any concerns, they can contact their caregivers using the in-app chat function.
  11. If desired, they can also choose to add or remove wearable devices and sensors.
  12. Lastly, they can check their profile, which shows personal information, settings and caregiver information.

How we built it

Software

• Frontend development using Flutter

• Backend development using FireBase

• Chatbot using Dialogflow

• Hosting and CICD setups using Netlify and Heroku and GitHub.

• SQL DB connected to the app with Restful API

• Google Colab notebooks to execute heavy GPU workloads and ML Algorithms

• Invision for developing WireFrames

• Figma for creating the prototype

Machine learning

We collected datasets from various sources such as Kaggle, JAFFE, and IMFDB and trained the machine learning model for a couple of tasks: the identification of emotions from facial expressions, identification of BMI from face images, identification of emotions from speech, and detection of falls from phone sensors. Determination of cardiovascular disease risk is also achieved by reviewing cohort studies and results in medical journals. After training the model, we deployed a demo of the emotion prediction model, BMI prediction model, and cardiovascular disease risk using Heroku service.

Challenges we ran into

It is difficult to find quality labelled data for training machine learning models, which in turn affects the accuracy rate. Given that this is a remote hackathon, we were also unable to test connection with wearables. While there is flexibility to use the app without external sensors, we plan to integrate with multiple wearable devices and platforms in the future.

Accomplishments that we are proud of

We have completed the whole UI design, developed a fully functional chatbot, trained machine learning models for emotions/ BMI prediction using facial images, and the backend & frontend development of the main features of the app.

What's next for medical intelligence applied (miia)

App Development

  1. Health data collection via speech recognition and wearables

  2. Data analytics dashboard

  3. In-app chat

Wearables

  1. Water-proof watch for seniors

  2. Water-proof necklace for seniors

Recruitment

We are planning to bring the project to the next stage. Shoot us a message if you're interested!

Built With

Share this project:

Updates