DeteX

Login
Register
Upload Image for Diagnosis
Save Diagnosis Form
Symptom based diagnosis
Map

Inspiration

The recent outbreak of the Covid-19 virus has left the world in a frenzy. The new virus has led health experts across the world to search for answers to combat this deadly sickness. Many of our friends and family have been affected personally by the virus. We made DeteX to help fight against Covid-19. Covid-19 and pneumonia both have similar symptoms. It is also shown that a patient contracting pneumonia soon after having Covid-19 is very common. Therefore, it is crucial that we can recognize which disease the patient has, or if they have both, so doctors can create a proper treatment plan for them. Covid-19 tests can take extended periods of time to give results, and can also contain false negatives and positives. Our project gives back a result as soon as the user gives their input, allowing for instantaneous results. It also stores all the information directly into the database This relieves pressure off of medical staff through higher efficiency of diagnosis, allowing them to have more time to treat patients, as well as having access to a database where they can view past diagnoses and filter through them.

What it does

Our web app uses Machine Learning to determine an X-ray as Viral Pneumonia, Bacterial Pneumonia, Covid-19, or Normal and has two main sections. One section is meant for medical professionals and the other can be used by normal patients. The section for medical professionals contains two pages.

When a user first opens the app they are prompted to log in. If they already are logged in, they will already be logged in and can view the pages, or they can register.

For medical professionals there is a diagnosis page and past diagnoses page. To view these pages you must be signed in with our firebase authentication as the past diagnoses page saves data to a specific user. On the diagnosis page, a user can upload an image. It will then get stored in our Firebase Storage and the user will be prompted with a button to get a diagnosis. The image will be sent from our Firebase Storage to our Google Cloud AutoML model and it will produce the chance of the image either being Viral Pneumonia, Bacterial Pneumonia, Covid-19, or Normal. This is super helpful as our diagnostics are very accurate, and relieves much time and pressure of extremely busy healthcare workers. After the user is given the predictions they are prompted with a form. The user can fill out the form with their patient’s name, date of birth, diagnosis, and the date. This will be sent to our Google Cloud Firestore and will be printed on the other page. The doctor can then go to the past diagnoses page where they can see the values they filled out on the form as well as the x-ray image they imputed. This is important as doctors won’t have to spend extra time documenting what happened, and there will be a much less chance of messing up because the doctor can see the diagnosis right in front of them. We also have a search filter which will be useful for when the diagnoses page gets very long. Also when the diagnosis form is filled out, the data is pushed to that account only, and a user will only be able to see the diagnoses they made.

The other section is meant for a common patient. This also has two pages. A user does not need to be logged in to view these pages. The first page is a diagnosis based on symptoms and the second is a map. A user can enter their symptoms and our algorithm will tell them if there is a chance if they have Pneumonia or Covid and what they should do. From here they can go to the map to find testing centers and hospitals near their location which is powered by the Google Maps and Places API.

How we built it

Overall we used Javascript, TensorflowJS, HTML, CSS, VueJS, and various Google Cloud APIs

VueJS:

Used Vue for the frontend
We decided to use Vue because of its two-way input binding
This makes it a lot easier to read and write data from Firebase
We used vanilla Javascript to make our symptom diagnosis algorithm

Google Cloud APIs and Services:

Firebase Authentication:
- Allows users to log in and save diagnoses to that specific user
- Makes sure you must be logged in to view the diagnose and past diagnoses page
Firebase Storage:
- Stores images to Firebase Storage when a user uploads them
- Used in the past diagnoses page to see the x-ray image for a past diagnosis
- Passes the image to the model so we don’t need to download images locally
- This makes the process much faster as the model get can the image from the cloud
AutoML Vision:
- Used Google Cloud's AutoML Vision API to train our machine learning model
- Used thousands of images of each type of pneumonia (from a Kaggle Dataset), Covid-19 (found data from GitHub repo), and a normal scan (from a Kaggle Dataset).
- Uses supervised learning to find patterns and content from our labeled images
Firebase Firestore:
- Used to store the results from the diagnose form and display them on the past diagnoses page
- Our script automatically grabs the x-ray image from the Firebase Storage and stores it in firestore, so then it displays the image, hyperlinked, with the rest of the results in the past diagnoses page
- Also used firestore querying for our search filter
Maps:
- Used Google Maps API for the Map on the Maps page. Displays all hospitals in a user’s location
Places:
- Used Google Places API to find the hospitals based on a users area and plot markers

Challenges we ran into

The biggest issue we faced was getting the image from Firestore into our AutoML model. We kept on permission errors when loading the image into the model, but after much debugging, reading through docs, and tweaking settings, we were able to combat the issue.

Accomplishments that we're proud of

This was our first time using VueJS as well as TensorflowJS, so integrating this was quite hard. However, we are proud of being able to create a fully functional web app at the end after all the debugging.

What we learned

We learned how to use TensorflowJS as well as Vue. We also used many Google Cloud APIs we've never used before which we learned a lot from.

What's next for Detex

Since flu season is coming up, we plan on adding a flu diagnosis to our model. We were not able to add it in this version of the project because we could not find data for flu x-ray scans to train our model on.