CovidScan-An AI Radiology Tool For COVID-19 Pandemic

CovidScan.ai is developed to be a secured AI platform with the purpose to assist radiologists with fast and accurate pneumonia dectection amid this COVID-19 pandemic.

Comment 13

Demo of web-app:

https://www.cv19scan.site/

(Please use Internet Explorer, or Firefox, our web-app currently doesn't support Chrome)

Dataset:

For the data analytics of COVID-19 pandemics, we used data collected by the Johns Hopkins University Center for Systems Science and Engineering updated on 4/30/2020.

For the chest X-ray detection models, we used combined 2 sources of dataset:

  1. The first source is the RSNA Pneumonia Detection Challenge dataset available on Kaggle contains several deidentified CXRs with 2 class labels of pneumonia and normal.

  2. The COVID-19 image data collection repository on GitHub is a growing collection of deidentified CXRs from COVID-19 cases internationally. The data is collected by Joseph Paul Cohen and his fellow collaborators at the University of Montreal

Eventually, our dataset consists of 5433 training data points, 624 validation data points and 16 test data points.

Inspiration

What will be working situation for medical staff in hospitals during and after the COVID-19 pandemic? How can the medical staff quickly and securely log in and perform PPE safety check while dealing with a huge influx of patients in critical conditions? How can we automate the process of COVID-19 diagnosis so precious time can be saved for both medical doctors and the patients? How can our solution for hospital later be scaled and implemented to be a essential tool for automating the daily operation at hospital even after the COVID-19 pandemics is over?

To answer these core questions, we did some background research to identify the main challenges in order to develop the best solutions around those:

COVID-19 Pandemic: Alt text Fig. 1: Map of Covid19 cases around the world (as of 4/30/2020). Our team created the map based on data collected by the Johns Hopkins University Center for Systems Science and Engineering.

As we see from the map above and the pie chart below, COVID-19, previously known as the novel Coronavirus, has killed more than 63,860 people and infected over 1,067,061 people in the United States alone, topping all other countries around the world. This number is continuing to grow every day.

Alt text

Fig. 2: Top 10 countries with most COVID-19 deaths.

The 3 main problems occur in the healthcare system during the pandemics are:

1. Confidentiality:

As you may see on the news, hospitals all over the U.S. (New York, Chicago,California…) and other countries (Italy, Spain…) are flooded with a huge influx of patients with critical conditions. With the increasing workload for the medical staff, patients’ confidential information may be put at risk if unauthorized personels can hack into the electronic medical record system. Thus, there is a need for a fast and secured method for medical staff to log in to the electronic medical record platform, so that the staff can move quickly with patients’ information inputting and still remain compliant with HIPPAA (Health Insurance Portability and Accountability Act). Badge scanning will be highly secured solution for this problem.

2. PPE Safety Check:

According to CDC, during COVID-19 pandemics, all healthcare workers should follow strict guidlines and protocols from OSHA regarding wearing PPE. All of the PPE prevents contact with the infectious agent, or body fluid that may contain the infectious agent, by creating a barrier between the worker and the infectious material. Gloves, protect the hands, gowns or aprons protect the skin and/or clothing, masks and respirators protect the mouth and nose, goggles protect the eyes, and face shields protect the entire face. N95 masks are the PPE most often used to control exposures to infections transmitted via the airborne route. Therefore, checking medical staff’s PPE safety protocol is especially crucial during this pandemics.

3. Long wait time for COVID-19 chest X-ray result: Alt text

Fig 3: Current chest X-ray diagnosis vs. novel process with CovidScan.ai

Patients can first be screened for flu-like symptoms using nasal swap to confirm their COVID-19 status. After 14 days of quarantine for confirmed cases, the hospital draws the patient’s blood and takes the patient’s chest X-ray. Chest X-ray is a golden standard for physicians and radiologists to check for the infection caused by the virus. An x-ray imaging will allow your doctor to see your lungs, heart and blood vessels to help determine if you have pneumonia. When interpreting the x-ray, the radiologist will look for white spots in the lungs (called infiltrates) that identify an infection. This exam, together with other vital signs such as temperature, or flu-like symptoms, will also help doctors determine whether a patient is infected with COVID-19 or other pneumonia-related diseases. The standard procedure of pneumonia diagnosis involves a radiologist reviewing chest x-ray images and send the result report to a patient’s primary care physician (PCP), who then will discuss the results with the patient. Alt text

_Fig 4: Chart of wait-time reduction of AI radiology tool (data from a simulation stud reported in Mauro et al., 2019). _

A survey by the University of Michigan shows that patients usually expect the result came back after 2-3 days a chest X-ray test for pneumonia. (Crist, 2017) However, the average wait time for the patients is 11 days (2 weeks). This long delay happens because radiologists usually need at least 20 minutes to review the X-ray while the number of images keeps stacking up after each operation day of the clinic. New research has found that an artificial intelligence (AI) radiology platform such as our CovidScan.ai can dramatically reduce the patient’s wait time significantly, cutting the average delay from 11 days to less than 3 days for abnormal radiographs with critical findings. (Mauro et al., 2019) With this wait-tine reduction, patients I critical cases will receive their results faster, and receive appropriate care sooner.

What it does

Using the power of pretrained machine learning models from open source, CovidScan.ai is created as a full-scaled AI tool for radiology clinics and hospitals. It can automate the process of security log-in, PPE safety check for medical staff and assist radiologists determine sign of COVID-19 on chest X-ray images with high accuracy indicates pneumonia. This tool of cutting edge technology can be used to reduce the workload for clinicians, and speed up patients’ wait time for pneumonia lab results in this critical time of the COVID-19 pandemic.

Alt text

Fig 5: Deployment process of pretrained ML model to the web-app

As explained in the figure above, the CovidScan web-app includes 3 main AI components:

1. ID Badge Scanner: For security purpose, only authorized personel can access to the web-app, which contains patients’ confidential health information (name, date of birth, chest X-ray, medical history…). Hence, the web-app will use pretrained scan the medical’s badge to grant them access to the software.

2. PPE Safety Check:

Due to hospitals/clinics’ strict guidelines in PPE usage, especially during this COVID-19 ourbreak, the web-app will ask the medical staff if he/she is in direct contact with patients for chest X-ray taking. If yes, then the web-app witll use AWS pretrained to check for medical staff’s PPE to see if the staff follow the safety protocols to minimize any exposures to the disease. If the medical staff passed both the secured check and safety, he/she can move on the the next step.

3. COVID-19 Chest X-ray Testing:

In the last step, the medical staff take patients’ chest X-ray images using the specialized machine and then upload the taken images to the database of web-app for testing for sign of COVID-19 infection or bacterial pneumonia. It is due to the fact that an AI system can review, highlight the pneumonia sign and classify each X-ray image all in less than 10 seconds (comparing the radiologist’s 20 minutes that we mentioned earlier), and it can do that same task effortlessly for 24 hours without taking a break. This time cut is especially critical in the time amid the pandemic of COVID-19. With this spreading rate, it will be overwhelming for radiologists to review a massive number of chest X-ray images of potential COVID-19 infected patients. With the assistance of CovidScan.ai, it can automatically highlight the suspected signs of pneumonia for the radiologists and speed up the process of chest X-ray review. Therefore, more COVID-19 positive-tested patients will get their result back faster and receive appropriate care sooner to prevent the spread of the virus.

How we built it

  1. Employee Badge Scanner: We developed this feature using the open-source python library Pyzbar. We have written the script in the JQuery which sends the snapshots from the live camera feed to the inference model at the backend. It can read one-dimensional barcodes and QR codes present on the employee’s ID badge. We implemented this feature to work with a snapshot of employees’ ID badge. Link: https://pypi.org/project/pyzbar/
  2. PPE Safety Check: We developed this feature using the open-source TensorFlow model for face mask detection. The backbone network only has 8 Conv layers and the total model has only 24 layers with the location and classification layers counted. The dataset is composed of WIDER Face and MAFA datasets. We have written the script in the JQuery which sends the snapshots from the live camera feed to the inference model at the backend. It works with live footage from any sort of cameras and detects people not wearing a face mask. Link: https://github.com/AIZOOTech/FaceMaskDetection
  3. Chest X-ray Classification: For this feature, we developed a Pytorch model. This project’s goal is to draw class activation heatmaps on suspected signs of pneumonia and then classify chest x-ray images as “Pneumonia” or “Normal”. For this project, we are going to use a dataset available at Kaggle consisting of 5433 training data points, 624 validation data points and 16 test data points. C. For the model, we load the pre-trained Resnet-152 available from Torchvision for transfer learning. ResNet-152 provides the state-of-art feature extraction since it is trained on a big dataset of ImageNet. ResNet-152, as the name sounds, consists of 152 convolutional layers. Due to its very deep network, the layers are arranged in a series of Residual blocks. These Residual blocks skip connections to help prevent the vanishing gradients which are a common problem with networks with deep architecture like ours. Resnet also supports Global Average Pooling Layer which is essential for our attention layer later on. For the attention layer to draw the heatmap, we use the global average pooling layer proposed in Zhou et al. Global average pooling layer explicitly enables the convolutional neural network (CNN) to have remarkable localization ability. We achieve 97% accuracy on the training dataset and 80% on the testing dataset.
  4. Web development: The trained weights of the deep learning models are deployed in a form of Django backend web app CovidScan.ai. While the minimal front-end of this web app is done using HTML, CSS, Jquery, Bootstrap. In our latter stage, the web-app will then be deployed and hosted on Debian server.

Technical Requirements:

The packages required for this project are as follows:

Torch (torch.nn, torch.optim, torchvision, torchvision.transforms)

Django

Numpy

Matplotlib

Scipy

PIL

Tensorflow

jQuery

Challenges we ran into

This hackathon project was a very different experience for us which challenged us throughout this project with the AWS sagemaker. This is the first time we all were working with AWS sagemaker and creating endpoints of the pre-trained TensorFlow model. Also, understanding curated models and determining their accuracy was a little bit challenging for us. Even after successfully deploying the model’s endpoints, calling Amazon SageMaker model endpoints using Amazon API Gateway and AWS Lambda gave us a very hard time.

Accomplishments that we're proud of

We manage to finish the project in such a limited time of 2 weeks in our free time from school and work. We still keep striving to submit on time while learning and developing at the same time. We are really satisfied and proud of our final product for the hackathon.

What we learned

Through this project, we learn to implement a complicated image-recognition deep learning models from AWS marketplace. We also learn the process of developing a mini data science project from finding dataset to training the deep learning model and finally deploy & integrate it into a web-app. This project can’t be done without the efforts and collaboration from a team with such diverse backgrounds in technical skills.

What's next for CovidScan:

In the next 2 months, our plan is:

  1. We will raise fund to invest more into the R&D process.

  2. We will partner with research lab to collect more dataset and find hospitals to test our solution. One of our memeber has published his newly collected dataset on this open-source github: https://github.com/nihalnihalani/COVID19-Detection-using-X-ray-images-/

  3. Regarding our R&D, we plan on improving the performance of the platform, preferably by reading more scientific literature on state-of-art deep learning models implemented for radiology.

  4. We also plan to add the bound box around the suspected area of infection on top of the heatmap to make the output image more interpretable for the radiologists. We are working to implament the multilabeling model of COVID-CXR on our dataset to improve our application. This model is published by The Artificial Intelligence Research and Innovation Lab at the City of London's Information Technology Services division and has accuracy 0.92, precision 0.5, recall 0.875, auc 0.96.

  5. In many pieces of literature, they mentioned developing the NLP model on radiology report with other structured variables such as age, race, gender, temperature... and integrating it with the computer vision model of chest X-ray to give the expert radiologist’s level of diagnosis. (Irvin et al., 2019; Mauro et al., 2019) We may try to implement that as we move further with the project in the future.

  6. With the improved results, we will publish these findings and methodologies in a user-interface journal so that it can be reviewed by expert computer scientists and radiologists in the field.

  7. Eventually, we will expand our classes to include more pneumonia-related diseases such as atelectasis, cardiomegaly, effusion, infiltration, etc. so that this platform can be widely used by the radiologists for general diagnosis even after the COVID-19 pandemics is over. Our end goal is to make this tool a scalable that can be used in all the radiology clinic across the globe, even in the rural area with limited access to the internet like those in Southeast Asia or Africa.

References:

Crist, C. (2017, November 30). Radiologists want patients to get test results faster. Retrieved from https://www.reuters.com/article/us-radiology-results-timeliness/radiologists-want-patients-to-get-test-results-faster-idUSKBN1DH2R6

Irvin, Jeremy & Rajpurkar, Pranav & Ko, Michael & Yu, Yifan & Ciurea-Ilcus, Silviana & Chute, Chris & Marklund, Henrik & Haghgoo, Behzad & Ball, Robyn & Shpanskaya, Katie & Seekins, Jayne & Mong, David & Halabi, Safwan & Sandberg, Jesse & Jones, Ricky & Larson, David & Langlotz, Curtis & Patel, Bhavik & Lungren, Matthew & Ng, Andrew. (2019). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.

Kent, J. (2019, September 30). Artificial Intelligence System Analyzes Chest X-Rays in 10 Seconds. Retrieved from https://healthitanalytics.com/news/artificial-intelligence-system-analyzes-chest-x-rays-in-10-seconds Lambert, J. (2020, March 11). What WHO calling the coronavirus outbreak a pandemic means. Retrieved from https://www.sciencenews.org/article/coronavirus-outbreak-who-pandemic

Mauro Annarumma, Samuel J. Withey, Robert J. Bakewell, Emanuele Pesce, Vicky Goh, Giovanni Montana. (2019). Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology; 180921 DOI: 10.1148/radiol.2018180921

Wang, L., & Wong, A. (2020, March 30). COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest Radiography Images. Retrieved from https://arxiv.org/abs/2003.09871

Built With

  • matplotlib
  • numpy
  • pil
  • pytorch1.0.1
  • torchvision0.2.2
+ 35 more
Share this project:

Updates