dr.ai

Inspiration

Healthcare is expensive, visiting hospitals is time-consuming and often requires creating extensive schedule changes in one's daily routine. Additionally, treatments often feel inadequate and lack personalization since Doctors keep incredibly busy. We figured we would develop a new AI-based approach to doctor's appointments, with the hope of making healthcare easily accessible at extremely low costs. dr.ai, who is a voice-powered virtual assistant, is able to gain a deep understanding of your medical needs in less than 5 minutes and provide incredible insights, details, and additional helpful utility to potentially change the way you think of your doctor's appointments in the future. The cool thing is that it significantly improves Houndify's "custom words" regex matching, and we're able to do this on a "custom domain".

What it does

Dr.ai has voice-powered conversation abilities that help her to develop a deep understanding of your symptoms. She uses this knowledge to determine/predict diseases you may develop along with their respective levels of severity/concern, help educate you about these conditions, and much more. Additionally, you can ask dr.ai to schedule an appointment with a real human doctor around the globe through a live, encrypted telemedicine video call in a matter of a few seconds! This gives you incredible power since you now have access to medical resources right in the palm of your hands, enabling you to lead a much healthier life.

How we built it

We built dr.ai using a variety of different tools:

We built a custom NLP layer surrounding Houndify's Speech-to-text and Text-to-speech API that can understand and interpret medical language. For this, we leverage clever statistical tests, math, and BioBERT, an extension of Google's BERT model specifically pre-trained on a corpus of medical data.
We implemented a second custom AI layer that uses scoring/ranking to determine the probability of a variety of different symptoms based on user input, where it has the capability to detect other symptoms you might be facing based on the ones you just described.
We use a combination of APIs and machine learning to predict what disease you may have, return more information about the disease itself, predict their severity, etc.
We're able to create a peer-to-peer encrypted video-sharing platform using Twilio to setup telemedicine calls with doctors across the world.

Challenges we ran into (need to explain these out)

Data Privacy: We wanted to ensure that Data Privacy was seriously considered. Where possible, we ensure data is never saved on the server, we disable client-side caching and ask for user consent before sending data to their telemedicine consultant. This required a lot of thought.
Smart Conversational AI/Conversational Workflows: We learned that designing for voice is difficult. You need to think of product design that feels more human-like, more intuitively, extremely comfortable and very natural. This required several iterations of getting the questions right so that the user has the flexibility to say a variety of different things, but we're still able to restrict the scope/domain where possible.
Integrating services together: We had a lot of different moving parts, from deep learning with state-of-the-art algorithms, to complex conversation workflows. This made it difficult to put all the parts together til lthe very end. Even though we had components built out, we couldn't add all of them into the app in time.
Houndify Speech-To-Text and Text-To-Speech APIs: We struggled a lot with the Houndify API, especially since we were working on a unique case of extending the scope of their API through NLP in a custom domain without the use of custom words. We also struggled with passing base64 encoded strings from the backend to the front, and writing reverse parsing logic from speech-to-text to text-to-speech.

Accomplishments that we're proud of

Were able to get a few parts of the project to work, and were able to get through ideation and completion. Especially great because 3/4 of us were beginners and we were able to make decent progress on a rather complex hack.

What we learned

Learned how to work better in a team, how to read documentation more efficiently, be more consistent with virtual environments, and about the challenges in medicine and natural language processing.

What's next for dr.ai

We're going to continue exploring natural language processing and conversational AI and its role in medicine. Hopefully, we can iron out our issues and see what professionals in the medical field think about the product itself.