A family member in the mental health field has seen first-hand that the industry is over-burdened and under-staffed. I felt that the right technology could offer a tool for patients to lessen their dependance on in-person care. It could also provide deeper and more regular insights to physicians that could enhance diagnosis and treatment.

What it does

MEL looks like a simple video diary, but it doesn't actually record video. Instead it analyzes the user's emotional state in real-time and then compares that to a baseline to determine if further action is warranted. In addition the user's speech is transcribed to text and analyzed on the Big Five Factor scale (Openness, Conscientiousness, Agreeableness, Extraversion, and Neuroticism). These records can also enhance a practitioner's interactions and assessment of the patient. If the patient is falling below their satisfactory baseline in these measurements, MEL offers some feedback on factors that can influence mental health such as amount of exercise and sleep. These values are pulled from the patient's HealthKit records. They are also given the option to contact an available mental health practitioner via an in-app video chat for further evaluation and recommendations.

How I built it

I used InterSystems IRIS for a backend data store, and pulled information on mental health care from a dataset provided by Humana. I built 2 native iOS apps - one for a patient and one for a practitioner. The MEL app uses some Core ML models to assess emotional state and analyze speech for the Big Five Factors. It uses ARKit to extract facial features such as the mouth and eye shape that are fed to one of the models to produce a real-time emotional state. It also uses real-time speech to text conversion in iOS to produce a transcript of the session. This is fed to 5 Core ML models to produce the Big Five Factors. This data and more are saved to IRIS, and can be retrieved and reviewed by a mental health practitioner.

MEL reduces the data to an overall score. If this value falls below an acceptable baseline, the app provides some insights to the user that can help them adjust behavior to improve their outlook. The information is pulled from the user's HealthKit records, which are filled by iOS and other health related apps. This includes minutes of daily exercise, amount of sleep, and sessions of mindful breathing. In addition MEL provides an overview of their recent medical sessions as provided by their insurance provider - Humana in this case. The patient is also offered the option to contact an available mental health professional via an in-app video chat to provide additional support and evaluation.

The second app, MEL_Provider is for mental health care providers. This implementation is much simpler than the MEL patient app, but in time could grow to a more comprehensive product. Currently the app provides a video chat interface for conducting a remote session with a patient. The app also pulls in the patient's IRIS data saved from the MEL app and displays it to the practitioner to give them a better overview of the patient's historical emotional state and personality assessment.

Challenges I ran into

This was a HUGE build and relied on everything going smoothly. Thankfully the vendors I used provided timely and consistent support to help understand their product, how to use it, and assisted with any setup and integration issues. I am very grateful for this support and could not have achieved as much without them. The biggest challenge I had was managing time, with a tremendous amount of code to write and new products to understand.

Accomplishments that I'm proud of

I'm most proud that the demonstration is complete end-to-end. Everything is functional and works from start to finish. I achieved all of my initial goals and then some.

What I learned

I learned a lot from the vendors, such as the complexity and depth of health and insurance records, and services necessary to manage them in a safe manner, yet still provide opportunities for third parties. I also learned about EHR exchanges and the various standards that support them.

What's next for MEL

While a great demonstration, there is a great deal of work to make MEL into a viable service. One area for further investigation is to match user profile assessments with appropriate and compatible mental health professionals. This could help tackle another issue in the industry where patients visit numerous professionals searching for one that works well for them.

Built With

  • agora
  • arkit
  • coreml
  • humana
  • intersystems
  • ios
  • iris
  • speech-to-text
  • swift
Share this project: