During these unprecedented times in the middle of a global pandemic, hospitals and clinics are reaching full capacity with an overflow of patients. Health care practitioners are forced to prioritize patients due to ICU limitations. In addition, it is inspired by our past experiences of visiting doctors. Often, we would meet/call the nurse to explain our symptoms before getting referred to a physician. If the nurse determines your symptoms to be severe enough, you get an appointment with a doctor. However, wait times can become tedious before your appointment. Therefore, we decided to develop an app that will streamline communication between patients and medical practitioners. This can assist in cutting down wait times, prioritizing patients with more severe symptoms, as well as giving users an easy way to track their symptoms.

What it does

A health app that allows users that are feeling sick to record and track symptoms along with general sentiment for the day. The recordings will be done via voice, and then converted to text automatically. Users are able to see/hear past recordings along with the text and will be able to fill out quick forms after each recording so the system can get a better idea of how they are feeling. Additionally, the app will run analysis algorithms on the user's data to determine the severity of this condition. This analysis can be will be used in two main ways: To create a prioritization algorithm that will rank the users on the system based on the severity of the condition. The reasoning for this is that a spike in COVID-19 has made medical attention much sparser and harder for most to get. Ranking the users would allow doctors/medical centers to better determine who needs care. Not only would they be able to see the rankings, but the analysis summary for each user would allow them to quickly determine who needs care. If the patient's condition is increasing in severity then we can connect them with a doctor to monitor their status because COVID-19 has made it much riskier for patients to go to the hospital. Converting the speak to text, and providing an updating summary on how the user is doing would make it faster for the doctor to review how they are doing and prescribe them appropriately.

About The Team

Our team is a group of computer programmers at various institutions of higher learning. We are made up of one sophomore, two juniors, and a senior. Our main goal for this project is to develop a working mobile app that addresses all the sectors of our idea and gives users a clean and efficient way to handle their medical needs. We are passionate about tech for social good and healthcare, and therefore we are eager to put our skills to use in these areas!

How we built it

First, we built the client side of our application using React Native to ensure that it has cross platform support. To address the backend and database needs, we used Google Firebase for User Authentication, Storage Space (for audio recordings and user profile pictures), and Cloud Firestore (scalable NoSQL database). Then we wrote and deployed a Google Cloud function to call the Google Cloud Voice to Text API to convert the users recording to text and display it to them in near real-time. Once we had this functionality working, we designed and implemented our custom text parsing algorithm to parse the user’s recording text for key symptoms. Then in the backend, we set up functionality to call the Symptom Checker API to fetch disease predictions based on the parsed symptoms, and store the symptoms and disease predictions in our database. For the functionality to send your symptoms to a doctor, we set up and deployed a cloud function to trigger the Twilio API to send a message to the specified number with the user information.

Challenges we ran into

There isn't much documentation on Twilio SMS for React Native Expo. Hence, there were many failed attempts before we succeeded. We also spent a long time finding an API or pre-trained ML model we could use to predict possible diseases from symptoms. It took us a while to figure out how to trigger the API calls. Initially, we had planned to use Tensorflow.js but none of their pre-trained models fit our use case. In addition, we spent a long time getting Google's voice to text function properly integrated with our application with the use of cloud functions.

Accomplishments that we're proud of

We are proud to have made a fully functioning mobile application with all the core features of our idea! Our application features accurate and instant Voice-to-Text, a custom text parsing model for symptom extraction, ability to store and track all previous recordings, easily set up a communication pipeline with your doctors, as well a profile screen to show a summary and disease predictions for all the user’s recordings.

What's next for HealthFirst AI

Improving the ML model for better predictions of diseases. Increasing our database of symptoms. Expanding the functionality of core features. Providing an AI chatbot and smoother communication pipeline with hospitals.

Award Categories

Best Health Hack - Our application addresses one of the most pressing health problems right now - overcrowded hospitals due to the COVID-19 pandemic. It allows users to easily track symptoms, provides ease of mind by setting up a communication pipeline to share status updates with doctors and family and provide accurate disease predictions based on described symptoms.

Best Use of Google Cloud - We used Google Firebase and Google Cloud Functions to handle the majority of our backend services for our mobile application. For Firebase, we used it for User Authentication, Firestore, and Storage Buckets. We used Google Cloud Functions to access the Voice to Text API to convert recordings to text, and also to integrate Twilio SMS API.

Best Use of Twilio API - We integrated the Twilio SMS API to send messages to doctors, family and friends. Users can add new contacts and send personalized text messages with the symptoms they are feeling.

Best Hack for Inclusion - Our voice to text feature for recording the symptoms ensures that everyone can easily use our product to get the medical attention they need. A lot of older patients and those that aren’t too familiar with technology can still use our product because they don’t have to manually type or select their symptoms. They can simply record, and it automatically converts it to text, runs symptom analysis, and disease predictions.

Best Pitch with Contrary- We created a more product-focused pitch that addresses problems currently in the market. In addition, we came up with a business model and did a competitor analysis. We also showcased our fully functioning MVP of our product idea.

+ 9 more
Share this project: