The four of us met at the Team Building Workshop when we got to SheHacks. None of had ideas for where we would go with a project. We found each other in a group focusing on Machine Learning and realized we had common interests in health and wellness. From there, the Audie came to be.

What it does

Audie's goal is for a patient to feel heard, but not intimidated by her patient experience. Audie is a bear that is activated by sound and records what the user says to her. The patient will talk about how she is feeling and Audie will transcribe her words into a sentiment value once a recording has been made. This value reflects how positive or negative what the patient says is. From here, a doctor can access the sentiment data of the patient and make an assessment on the patient's progress.

How we built it

Audie consists of three major components. The first component is the speech component. When someone speaks to the bear, a microphone will pick up the audio and record it to a file. We used an Arduino board and it's sensors to create the microphone set-up. The audio file is generated with MATLAB. The second component of the Audie project is the encoding and analysis of the data. We used the Google Cloud Speech API to take the audio file and transcribe to text. From there, the Google Cloud Natural Language Processing API is used with the text to calculate the sentiment of the patient. The data from the sentiment calculation is stored. The last component is the data visualization. We want a doctor to be able to look at the whole set of data for a patient, not just a particular day's score. The data points from the same patient are are all put on a line-graph, so a doctor can track the patients feelings over time.

Challenges we ran into

All four of us made an effort to challenge ourselves these past 36 hours. Most of the challenges we faced were worked through; we configured the Arduino board, and learned how the Google APIs worked, but we struggled to get the full visualization feature we want for the Audie. The original idea was to have a website that doctors could reach with an option for them to refresh the data. After having issues with HTML configuration with Python, we decided to work on a line graph made with Python. Unfortunately, this is less user-friendly that we had hoped for, but with more time we would have figured out an option for a website.

Accomplishments that we're proud of

We are really excited as a group of mostly beginner hackers to have accomplished what we did. All four of us come from varying backgrounds and coding ability, but we used what we know to teach one another and leverage our hacking ability. We are most excited about our ability to integrate the Google Cloud APIs and an interactive toy. We think it is really excited that Audie isn't just a program working on a computer, but a tangible product.

What we learned

One of the most exciting parts of Audie for us is the Google Cloud APIs we used. Had we not gone to one of the first workshops on Google APIs, Audie would not have worked nearly as well as it is now. Before this hackathon, APIs seemed incredibly abstract, but after working with them so closely the past couple days, we all feel that we know how they work and how to use them. This is not only helpful for our project, but is great for understanding how APIs work in all the applications we use.

What's next for Audie

Firstly, a more powerful sound sensor and an integrated microphone would be helpful. Right now, the Arduino sound sesnor we are using is not quite as reliable as one would need to accurately trigger the recording of audio files, and we use the computer Audie is tethered to as a microphone. Most importantly, we would like extend Audie's capabilities. What's really empowering about creating the Audie, is that it has the capacity to help so many people. Our intention is that the Audie be marketed towards hospitals and their work with children who frequently visit. It is very stressful for a child to come into a hospital, and sometimes a child will be hesitant acknowledge how she truly feels. Audie is a tool that can be used at home and in a hospital to monitor a child's symptoms more continuously. We would really like to see how Audie can grow as well. The next step would be to implement a Machine Learning Algorithm to pick up on symptom keywords that will either help diagnose a patient, mark when a patient is experiencing a flare-up, or notify a doctor when there is a new development in the patient's prognosis.

Built With

Share this project: