-
-
Fig 1: Current chest X-ray diagnosis vs. novel process with PneumoScan.ai
-
Fig 2: Deployment process of pretrained ML models to the web-app
-
Fig 3: Developing team
Inspiration
COVID-19 Pandemic:
As a live update from CNN, COVID-19, previously known as novel Coronavirus, has killed more than 108,167 people and infected over 1,765,030 people in 144 countries. (Pettersson et al., 2020) This number is continuing to grow every day. In the United States, the COVID has been spreading rapidly. Currently, 21 states have declared an emergency over the spread of Coronavirus and they reported over 1,000 confirmed cases, and over two dozen fatalities attributed to COVID-19. The 3 main problems occur in the healthcare system during the pandemics are:
1. Confidentiality:
As you may see on the news, hospitals all over the U.S. (New York, Chicago,California…) and other countries (Italy, Spain…) are flooded with a huge influx of patients with critical conditions. With the increasing workload for the medical staff, patients’ confidential information may be put at risk if unauthorized personels can hack into the electronic medical record system. Thus, there is a need for a fast and secured method for medical staff to log in to the electronic medical record platform, so that the staff can move quickly with patients’ information inputting and still remain compliant with HIPPAA (Health Insurance Portability and Accountability Act). Badge scanning will be highly secured solution for this problem.
2. PPE Safety Check:
According to CDC, during COVID-19 pandemics, all healthcare workers should follow strict guidlines and protocols from OSHA regarding wearing PPE. All of the PPE prevents contact with the infectious agent, or body fluid that may contain the infectious agent, by creating a barrier between the worker and the infectious material. Gloves, protect the hands, gowns or aprons protect the skin and/or clothing, masks and respirators protect the mouth and nose, goggles protect the eyes, and face shields protect the entire face. N95 masks are the PPE most often used to control exposures to infections transmitted via the airborne route. Therefore, checking medical staff’s PPE safety protocol is especially crucial during this pandemics.
3. Long wait time for COVID-19 chest X-ray result:
Fig 1: Current chest X-ray diagnosis vs. novel process with PneumoScan.ai
Patients can first be screened for flu-like symptoms using nasal swap to confirm their COVID-19 status. After 14 days of quarantine for confirmed cases, the hospital draws the patient’s blood and takes the patient’s chest X-ray. Chest X-ray is a golden standard for physicians and radiologists to check for the infection caused by the virus. An x-ray imaging will allow your doctor to see your lungs, heart and blood vessels to help determine if you have pneumonia. When interpreting the x-ray, the radiologist will look for white spots in the lungs (called infiltrates) that identify an infection. This exam, together with other vital signs such as temperature, or flu-like symptoms, will also help doctors determine whether a patient is infected with COVID-19 or other pneumonia-related diseases. The standard procedure of pneumonia diagnosis involves a radiologist reviewing chest x-ray images and send the result report to a patient’s primary care physician (PCP), who then will discuss the results with the patient.
A survey by the University of Michigan shows that patients usually expect the result came back after 2-3 days a chest X-ray test for pneumonia. (Crist, 2017) However, the average wait time for the patients is 11 days (2 weeks). This long delay happens because radiologists usually need at least 20 minutes to review the X-ray while the number of images keeps stacking up after each operation day of the clinic. Patients usually have to frequently call in with their clinic to check for the result after 5 days of not hearing back. With the patients’ wait time in minds, radiologists want patients to get test results faster, so that patients with positive pneumonia lab results can receive appropriate care sooner.
What it does
Using the power of pretrained machine learning models from AWS Marketplace and other AWS services, PneumoScan.ai is created as a full-scaled AI tool for radiology clinics and hospitals. It can automate the process of security log-in, PPE safety check for medical staff and assist radiologists determine sign of COVID-19 on chest X-ray images with high accuracy indicates pneumonia.This tool of cutting edge technology can be used to reduce the workload for clinicians, and speed up patients’ wait time for pneumonia lab results in this critical time of the COVID-19 pandemic.
Fig 2: Deployment process of pretrained ML model to the web-app
As explained in the figure above, the PneumoScan web-app includes 3 main AI components:
1. ID Badge Scanner: For security purpose, only authorized personel can access to the web-app, which contains patients’ confidential health information (name, date of birth, chest X-ray, medical history…). Hence, the web-app will use pretrained scan the medical’s badge to grant them access to the software.
2. PPE Safety Check:
Due to hospitals/clinics’ strict guidelines in PPE usage, especially during this COVID-19 ourbreak, the web-app will ask the medical staff if he/she is in direct contact with patients for chest X-ray taking. If yes, then the web-app witll use AWS pretrained to check for medical staff’s PPE to see if the staff follow the safety protocols to minimize any exposures to the disease. If the medical staff passed both the secured check and safety, he/she can move on the the next step.
3. COVID-19 Chest X-ray Testing:
In the last step, the medical staff take patients’ chest X-ray images using the specialized machine and then upload the taken images to the database of web-app for testing for sign of COVID-19 infection or bacterial pneumonia. New research has found that an artificial intelligence (AI) radiology platform such as our PneumoScan.ai can dramatically reduce the patient’s wait time significantly, cutting the average delay from 11 days to less than 3 days. (Mauro et al., 2019) It is due to the fact that an AI system can review, highlight the pneumonia sign and classify each X-ray image all in less than 10 seconds (comparing the radiologist’s 20 minutes that we mentioned earlier), and it can do that same task effortlessly for 24 hours without taking a break. This time cut is especially critical in the time amid the pandemic of COVID-19. With this spreading rate, it will be overwhelming for radiologists to review a massive number of chest X-ray images of potential COVID-19 infected patients. With the assistance of PneumoScan.ai, it can automatically highlight the suspected signs of pneumonia for the radiologists and speed up the process of chest X-ray review. Therefore, more COVID-19 positive-tested patients will get their result back faster and receive appropriate care sooner to prevent the spread of the virus.
How we built it
1. Employee Badge Scanner:
We developed this feature using the Quantipil’s Barcode/QR-code Scanner (curated model) from AWS Marketplace. We implemented this model to work with snapshot of employees’ ID badge.
Link: https://aws.amazon.com/marketplace/pp/Quantiphi-BarcodeQR-code-Scanner/prodview-avylkh3xuqs3e
2. PPE Safety Check:
We developed this feature using the VitechLab’s PPE Detector for Laboratory Safety (curated model) from AWS Marketplace. We implemented this model on Sagemaker to work with live video. The model is trained on the dataset manually selected and annotated by the VITechLab team. It works with live footage from CCTV cameras and detects people not wearing any of four objects: Coat, Glasses, Glove, Mask.
Link: https://aws.amazon.com/marketplace/pp/prodview-b53upp27dnmzq?ref_=_ml_hackathon
3. Chest X-ray Classification:
For this feature, we used the COVID-Net Large, a pretrained deep convolutional neural network design tailored for the detection of COVID-19 cases from chest radiography images. (Wang, 2020) This pretrained model is developed by University of Waterloo and Darwin lab and they made it open-source for the public to implement. The model is trained on COVIDx dataset, which is comprised of 16,756 chest radiography images across 13,645 patient cases from two open access data repositories. This model can classify the chest X-ray images into 3 classes: COVID-19, bacterial pneumonia, and normal. The model achieve a 90% sensitivity on COVID-19 classification. More detailed metrics can be found in the confusion matrix and classification reports below:
Link: https://github.com/lindawangg/COVID-Net
Web development:
The core backend of the pneumoscan web app is a python based framework - Django. For the client-side JQuery is used for the seamless user experience. The major data-driven events such as contacting APIs developed on AWS API Gateway are kept asynchronous with the help of JQuery.
Deployment:
The application was deployed in ElasticBeanstalk primarily for the below reasons:
Support of Python Platform
Seamless Deployment for application updates
Robust Monitoring with CloudWatch logs
High Availability
Elastic Beanstalk helps in providing the environment for developing and deploying without worrying about nitty-gritty details
Technical Requirements:
The packages required for this project are as follows:
TensorFlow
Numpy
Matplotlib
Scipy
PIL
AWS services:
Marketplace
S3
Sagemaker
ElasticBeanstalk
AWS Lambda
AWS API Gateway
Challenges we ran into
This hackathon project was a very different experience for us which challenged us throughout this project with the AWS sagemaker. This is the first time we all were working with AWS sagemaker and creating endpoints of the pre-trained TensorFlow model. Also, understanding curated models and determining their accuracy was a little bit challenging for us. Even after successfully deploying the model’s endpoints, calling Amazon SageMaker model endpoints using Amazon API Gateway and AWS Lambda gave us a very hard time.
Accomplishments that we're proud of
We manage to finish the project in such a limited time of 2 weeks in our free time from school and work. We still keep striving to submit on time while learning and developing at the same time. We are really satisfied and proud of our final product for the hackathon.
What we learned
Through this project, we learn to implement a complicated image-recognition deep learning models from AWS marketplace. We also learn the process of developing a mini data science project from finding dataset to training the deep learning model and finally deploy & integrate it into a web-app. This project can’t be done without the efforts and collaboration from a team with such diverse backgrounds in technical skills.
Fig 3: Developing Team
What's next for PneumoScan: An AI Tool For PPE Check & COVID19 Testing
- AWS does provide a great variety of prepackaged ML models for different daily operation tasks that can be incorporated in a radiology clinics and hospitals. In our next step, we want to implement and include the following curated models (that we didn’t have enough time to explore yet):
a) 7Park Drug Name ENR to automate the process of drug name extraction from radiology report.
b) 7Park Transaction Data Parsing (NER) to extract information credit cards or health insurance card for faster payment processing.
These two features will help our Pneumoscan platform moving on to the next level of becoming a go-to multifunctional tool for radiology clinic and hospital.
2.. After this competition, we will work on improving the performance of the platform, preferably by reading more scientific literature on state-of-art deep learning models implemented for radiology.
We also plan to add the bound box around the suspected area of infection on top of the heatmap to make the output image more interpretable for the radiologists.
In many pieces of literature, they mentioned developing the NLP model on radiology report with other structured variables such as age, race, gender, temperature... and integrating it with the computer vision model of chest X-ray to give the expert radiologist’s level of diagnosis. (Irvin et al., 2019; Mauro et al., 2019) We may try to implement that as we move further with the project in the future.
With the improved results, we will publish these findings and methodologies in a user-interface journal so that it can be reviewed by expert computer scientists and radiologists in the field.
Eventually, we will expand our classes to include more pneumonia-related diseases such as atelectasis, cardiomegaly, effusion, infiltration, etc. so that this platform can be widely used by the radiologists for general diagnosis even after the COVID-19 pandemics is over. Our end goal is to make this tool a scalable that can be used in all the radiology clinic across the globe, even in the rural area with limited access to the internet like those in Southeast Asia or Africa.
References:
Crist, C. (2017, November 30). Radiologists want patients to get test results faster. Retrieved from https://www.reuters.com/article/us-radiology-results-timeliness/radiologists-want-patients-to-get-test-results-faster-idUSKBN1DH2R6
Irvin, Jeremy & Rajpurkar, Pranav & Ko, Michael & Yu, Yifan & Ciurea-Ilcus, Silviana & Chute, Chris & Marklund, Henrik & Haghgoo, Behzad & Ball, Robyn & Shpanskaya, Katie & Seekins, Jayne & Mong, David & Halabi, Safwan & Sandberg, Jesse & Jones, Ricky & Larson, David & Langlotz, Curtis & Patel, Bhavik & Lungren, Matthew & Ng, Andrew. (2019). CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison.
Kent, J. (2019, September 30). Artificial Intelligence System Analyzes Chest X-Rays in 10 Seconds. Retrieved from https://healthitanalytics.com/news/artificial-intelligence-system-analyzes-chest-x-rays-in-10-seconds Lambert, J. (2020, March 11). What WHO calling the coronavirus outbreak a pandemic means. Retrieved from https://www.sciencenews.org/article/coronavirus-outbreak-who-pandemic
Mauro Annarumma, Samuel J. Withey, Robert J. Bakewell, Emanuele Pesce, Vicky Goh, Giovanni Montana. (2019). Automated Triaging of Adult Chest Radiographs with Deep Artificial Neural Networks. Radiology; 180921 DOI: 10.1148/radiol.2018180921
Wang, L., & Wong, A. (2020, March 30). COVID-Net: A Tailored Deep Convolutional Neural Network Design for Detection of COVID-19 Cases from Chest Radiography Images. Retrieved from https://arxiv.org/abs/2003.09871
Built With
- amazon-web-services
- django
- elasticbeantalk
- lightsail
- sagemaker
- tensorflow
Log in or sign up for Devpost to join the conversation.