VoiceMD

• 

  Dashboard

Inspiration

We wanted to develop a tool that people could use in developing countries where medical access is heavily restricted and a tool that could help doctors get a better assessment of the patient before they request an appointment.

What it does

The web app asks the user to complete a few tasks and say a few phrases. From these phrases we use the Houndify API to display speech to text and determine the meaning of the phrase that the person is saying. Then it determines displays a medical dashboard with emotion, the pitch, jitter, and shimmer of your voice, and the probability of having diseases like Parkinson's, depression, etc. The Alexa skill asks the user if they are in pain and gives a general statement as to what is most likely the cause of that pain.

How we built it

We build it using flask where we could have the user do a few tasks and record their voice. Using the Houndify API we were able to convert this text to speech in order to give a visualization to the user. We also used the Houndify API to determine the meaning of what the user is saying. Then we ran this audio through a python script using praat in python to get the pitch, jitter, shimmer and a few other attributes from each audio clip. On the medical diagnostic dashboard we used a custom python package to determine the emotion of the user. We also display the pitch, jitter and the shimmer which was calculated by our praat python script. We also determine the percentage chance of having diseases such as depression, Parkinson's, Alzheimer's, etc. We hosted the medical diagnostic page on Google Cloud.

Using python and Pytorch we created a feed forward neural net trained a model to determine if, based on the pitch, shimmer, and jitter, the probability that the user had a certain disease. The neural net had a loss of about 0.54 and the accuracy of this model ranges from 60% to 70%. Here is one data set that we used: https://archive.ics.uci.edu/ml/datasets/Parkinson+Speech+Dataset+with++Multiple+Types+of+Sound+Recordings#

We used Voiceflow to create an Alexa skill that we used to ask more general medical questions and give users a general reply. This Alexa skill will ask the user their name and will ask them what body part they feel pain in. After getting that information the Alexa will respond with the most common reason for pain in that part of the body.

Challenges we ran into

Determining how to detect disease, in order to detect disease we trained our model on a data set that had the pitch, jitter, and shimmer of people's voices if they had a disease like Parkinson's. Working with Alexa, initially we wanted to use Alexa as the voice assistant the people could talk to, however we discovered that there is currently no way to record audio on the Alexa. So we switched the focus of the Alexa to be a general purpose medical evaluator.

Accomplishments that we're proud of

We are proud to have build an intuitive app with a great UI that has the potential to help people in developing countries and help doctors aid a larger number of patients with greater efficiency.

What we learned

In the past we have worked with Tensorflow to build neural networks and machine learning models, however as we were working with python we decided to try using Pytorch as it seemed like a much more versatile tool for python. Working with audio files and manipulating them to get data from them was an interesting challenge that we loved learning from. We learned how to program with Alexa and how to customize commands through Voiceflow.

What's next for VoiceMD

Polish the webapp so that it is ready for production. Launch the site and market it as a solution to aid in the decision to go see a doctor. Publish the Alexa skill so that people can use it to self diagnose the pain that they are having.

Built With

• amazon-alexa
• flask
• google-cloud
• houndify
• html5
• javascript
• praat
• python
• pytorch
• soundhound
• voiceflow