Here are some quick links to some of the resources we developed while creating our project:
As our population ages we will begin to have a lot of multimorbidities. The aging population will have higher rates of diabetes, hypertension, and other chronic ailments. Mobile health (mHealth) platforms using smartphones have proven effective for monitoring blood pressure, glucose and other health related symptoms. However, applications are not always accessible for the elderly population. Finger sensitivity and mobility can be an obstacle for the elderly as it impairs their ability to interact with apps. Features such as larger font size, high contrast, and text to speech functionality are often neglected due to the lieu of modern design trends intended to appeal to younger audiences.
We designed our app, miia (Medical Intelligently Applied) to be accessible and usable by most seniors. Miia is an application that will help track and manage health conditions for the elderly population. For instance, we implemented a Chatbot function to help seniors input their vital signs. The chatbot can be made to speak aloud, while the senior can utilize their voice which is then converted to text. The chatbot can also ask questions to monitor symptoms and mood to screen for infection or depression, respectively. Furthermore, our app will track mobility and activity functions of our users through drawing data from the built-in accelerometer, gyroscope, and other smartphone sensors. This will help us predict activity level and potentially prevent frailty and traumatic falls with seniors.
How to use miia
Miia can be used through entering https://miia.me/ and signing in with gmail or by creating a new account. Once you've logged into miia you're greeted by the main dashboard that provides an overview of your profile along with several different tabs. Here users can chat with miia, sync wearables, and receive diagnostic reports from health checkups. Current functionality of the application is limited to conducting conversations with the chatbot while also completing facial recognition scans that detect mood and BMI.
Nonetheless, our current figma prototype serves as a better representation of the apps final functionality and design. In contrast to the web application the prototype is developed for mobile devices to better serve the elderly through prioritizing convenience and mobility. The prototype itself is fully interactive as users have the ability to click, scroll and drag through both caregiver and patient interfaces.
What it does
The system leverages AI technology to analyze data collected from facial recognition, speech recognition, wearable devices and/or IoT on a daily basis, and alert the caregivers if there is any identified risks. The platform also provides a way to facilitate communication between caregivers and care recipients, while aiding with health management to alleviate caregiver stress.
Health data collection
We ensure the health data collection process is easy to follow by having the whole health check up process guided by our AI chatbot miia, which include the following:
Facial recognition - facial image taken for analysis of cardiovascular diseases risks, emotions, BMI and etc.
Speech recognition - speech recorded and analyzed for emotions and mood
AI chatbot - collect health data unavailable in facial and speech recognition/ wearable devices
Phone sensors - detection of fall
Wearable devices/sensors - measurements including but not limited to blood pressure/ heart rate/ sleeping pattern/ activity
Elderly focus design
Voice control - elderly users can choose to interact with chatbot by voice or text
AI Chatbot to stimulate human interactions
Enlarged text and other accessibility features
Reminder system - visual and sound alerts can be snoozed until the elderly login and complete the health monitoring daily
Data visualization for caregivers
Data analytics dashboard - show key metric of elderly over one month
Detailed health reports of elderly - details of each health parameter
Alert system for identified issues - caregivers can set threshold values according to elderly's condition; red warning symbols and notification pop up when value above/ below normal
- Signs up in the app and makes a profile for both themselves and their care recipient.
- After choosing the caregiver option, they will set up an account with their email and phone number, and set a password.
- Then, the caregiver will add the patient’s name and phone number.
- They can then add the pre-existing medical conditions of their care recipient. In this case, the preset conditions are common chronic diseases but there is also the option to add more conditions and background information.
- The caregiver can choose important metrics to monitor for certain chronic conditions, such as blood sugar level for diabetes, or mood for depression.
- After adding the background information for the patient, a unique pin will be generated for connecting the caregiver with the care recipient.
- A confirmation screen will also show the patient’s conditions and metrics to follow. If there are multiple care recipients, the caregiver can add another patient.
- On a daily basis, caregivers log in and monitor health of care recipients, the most important metric on display. The red notification symbol indicates a warning that requires caregivers to follow up on a metric.
- In the patient profile, the caregiver can change or add more metrics to monitor, chat with the patients, or edit the patient profiles.
Elderly/ Care Recipient
- Care recipient received a text message from the caregiver with his/ her unique pin. If a senior is unfamiliar with technology, the caregiver can help him/ her to set up the app.
- Choose to sign up as a patient, and enter the pin received.
- Our chatbot, guide seniors through the whole health checkup process on a daily basis
- The patient can choose to text or speak to the chatbot.
- Miia will proceed to initiate the process of health check by taking their facial image
- Miia will first ask a few questions regarding their physical and mental health, such as body temperature, blood pressure, or mood and the senior can input manually or tell miia their measurements. For voice inputs, Miia will repeat the measurement to verify.
- Depending on the needs of the senior and caregiver, the chatbot can also ask about other metrics, give reminders, or chat with the senior.
- After health check, users will be redirected to a health overview which summarizes results for the senior.
- Key metrics of seniors are shown in measurements. If the user is interested in knowing more of a particular metric, they can click the metric and look into the details.
- If seniors have any concerns, they can contact their caregivers using the in-app chat function.
- If desired, they can also choose to add or remove wearable devices and sensors.
- Lastly, they can check their profile, which shows personal information, settings and caregiver information.
How we built it
• Frontend Dev using Angular, FireBase Authentication.
• Node Libraries Likes charts.js PWAs, BootStrap, Material Design, etc.
• Hosting and CICD setups using Netlify and Heroku and GitHub.
• Domain and SSL certificate from Namecheap and Let's Encrypt.
• SQL DB connected to the app with Restful API.
• Google Colab notebooks to execute heavy GPU workloads and ML Algorithms.
• Invision for developing WireFrames
• Figma for creating final prototype
• Slack for Internal Communications & Google Drive for Documents, Images, etc.
We collected datasets from varies sources such as Kaggle, JAFFE and IMFDB and trained the machine learning model for a couple of tasks: the identification of emotions from facial expressions, identification of BMI from face images, identification of emotions from speech, and detection of falls from phone sensors. Determination of cardiovascular disease risk is also achieved by reviewing cohort studies and results in medical journals. After training the model, we deployed a demo of the emotion prediction model, BMI prediction model, and cardiovascular disease risk using Heroku service.
Challenges we ran into
It is difficult to find quality labelled data for training machine learning models, which in turn affects the accuracy rate. Given that this is a remote hackathon, we were also unable to test connection with wearables. While there is flexibility to use the app without external sensors, we plan to integrate with multiple wearable devices and platforms in the future.
To facilitate the adoption of our technology, we plan to target caregivers (B2B) as our primary target demographic. Currently there are 34 million caregivers for the elderly in the United States, with 5 million of them being long distance caregivers. Our goal is to introduce our product, while increasing our adoption rate, and thus solidify our application as an essential tool for caregivers worldwide.
Currently miias distributions channels will be limited to mobile app stores found on both android and ios devices. In later iterations miia will transition to being available as a web application for desktops.
Our go-to-market strategy during distribution will include a combination of freemium and viral approaches. This in-turn provides us with financial incentives for early adopters, who are able to take advantage of the 2-month free trial while having the ability to subscribe later. We’d also like to introduce a referral system where users are able to promote our application while being rewarded for successful signups. In addition to this, we aim to partner with health organizations (clinics/ hospital/ national health insurance) alongside deploying through-the-line marketing tactics in order to enhance customer reach and maximize customer acquisition.
What's next for miia!
Health data collection via speech recognition and wearables
Data analytics dashboard
Water-proof watch for seniors
Water-proof necklace for seniors
We are planning to bring the project to the next stage. Shoot us a message if you're interested!