Healthily

main screen
account info
patient records form
appointment booking

Inspiration

There are many moments in our life where we feel sick, and more often than not, the idea of looking forward for a doctor appointment to gain valuable diagnosis and treatment options can be quite overwhelming. Thats where we though, "what if we could harness the power of AI and large language models in order to make a type of virtual doctor, which could provide you instant possible diagnosis and smoothen the process of appointment scheduling?

Thats when we though of the idea of Healthily

What it does

Healthily is an app which allows people to communicate with a personalized AI doctor providing a natural conversation flow that would typically occur between a doctor and a patient at a health checkup. Healthily provides rapid and sensible recommendations as well as possible diagnosis for a user's health problems. After the conversation, the AI then allows the user to book an appointment with their family doctor, sending the chat history to the doctor to aid in the appointment as well.

Now imagine not having to fill a doctor's visit form every checkup?

How we built it

We built it using a GPT-3.5 text completion, fine tuned on medical data. We also made use of ChatGPT whisper api which can convert speech to text, and google text to speech api for converting the speech back to text. The frontend was built using React Native with Expo, and utilizing Expo Audio for recording audio and playing sound back to the app.

Challenges we ran into

Finding the right speech to text api was a problem because google's speech to text api wasn't really that accurate, in which we ended up having to utilize OpenAI Whisper API. Furthermore, we encountered problems with prompt engineering the GPT-3.5 model in order to get the right output.

Accomplishments that we're proud of

We're proud of being able to implement speech to text as well as text to speech. Furthermore, being able to implement audio recording on an expo app using Expo Audio was a huge plus. Also, Frontend Engineering the mobile app in such a way to bring up modals on user speech input as very cool.

What we learned

We learned about how to prompt engineer the GPT-3.5 model to steer it towards the responses we need, as well as fine-tuning the model in order to ensure that it gave accurate answers. Furthermore, we also learned very important React hooks, useState and useEffect, as well as Expo libraries including Expo Audio, and dealing with Audio stream data.