FACO: Fight Against Corona

GIF
Confusion matrix for our final model

INSPIRATION

A diagnosis of respiratory disease is one of the most common outcomes of visiting a doctor. Respiratory diseases can be caused by inflammation, bacterial infection or viral infection of the respiratory tract. Diseases caused by inflammation include chronic conditions such as asthma, cystic fibrosis, COVID-19, and chronic obstructive pulmonary disease (COPD). Acute conditions, caused by either bacterial or viral infection, can affect either the upper or lower respiratory tract. Upper respiratory tract infections include common colds while lower respiratory tract infections include diseases such as pneumonia. Other infections include influenza, acute bronchitis, and bronchiolitis. Typically, doctors use stethoscopes to listen to the lungs as the first indication of a respiratory problem. The information available from these sounds is compromised as the sound has to first pass through the chest musculature which muffles high-pitched components of respiratory sounds. In contrast, the lungs are directly connected to the atmosphere during respiratory events such as coughs, heart rate.

PROBLEM STATEMENT

In this difficult time, a lot of people panic if they have signs of any of the symptoms, and they want to visit the doctor.
It isn’t necessary for the patients to always visit the doctor, as they might have a normal fever, cold or other condition that does not require immediate medical care.
The patient who might not have COVID-19 might contract the disease during his visit to the Corona testing booth, or expose others if they are infected.
Most of the diseases related to the respiratory systems can be assessed by the use of a stethoscope, which requires the patient to be physically present with the doctor.
Healthcare access is limited—doctors can only see so many people, and people living in rural areas may have to travel to seek care, potentially exposing others and themselves.

SOLUTION

We provide a point of care diagnostic solutions for tele-health that are easily integrated into existing platforms. We are working on an app to provide instant clinical quality diagnostic tests and management tools directly to consumers and healthcare providers. Our app is based on the premise that cough and breathing sounds carry vital information on the state of the respiratory tract. It is created to diagnose and measure the severity of a wide range of chronic and acute diseases such as corona, pneumonia, asthma, bronchiolitis and chronic obstructive pulmonary disease (COPD) using this insight. These audible sounds, used by our app, contain significantly more information than the sounds picked up by a stethoscope. app approach is automated and removes the need for human interpretation of respiratory sounds, plus user disease can also be detected by measuring heart beat from camera of smartphone.

The application works in the following manner:

User downloads the application from the app store and registers himself/herself.
After creating his/her account, they have to go through a questionnaire describing their symptoms like headache, fever, cough, cold etc.
After the questionnaire, the app records the users’ coughing, speaking, breathing and heart rate in form of video from smartphone.
After recording, the integrated AI system will analyze the sound recording, heart rate comparing it with a large database of respiratory sounds. If it detects any specific pattern inherent to a particular disease in the recording, it will enable the patient to contact a nearby specialist doctor.
The doctor then receives a notification on a counterpart of this app, for doctors. The doctor can view the form, watch the audio recording, and also read the report given by the AI of the application.
The doctor, depending upon the report of the AI, will develop a diagnosis, suggest medicines, or recommend a hospital visit if the person shows symptoms of corona or other serious condition.
In cases where the AI detects a very seriously ill patient, it will also enable the physician to call an ambulance to the users’ location and continuously track the user.

HOW WE ARE GOING TO BUILD IT

We will take a machine learning approach to develop highly-accurate algorithms that diagnose disease from cough and respiratory sounds. Machine learning is an artificial intelligence technique that constructs algorithms with the ability to learn from data. In our approach, signatures that characterize the respiratory tract are extracted from cough and breathing sounds. We start by matching signatures in a large database of sound recordings with known clinical diagnoses. Our machine learning tools then find the optimum combination of these signatures to create an accurate diagnostic test or severity measure (this is called classification). Importantly, we believe these signatures are consistent across the population and not specific to an individual so there is no need for a personalized database Following are the steps the app will take:

Receive an audio signal from the user's phone microphone
Filter the signal so as to improve its quality and remove background noise
Run the signal through an artificial neural network which will decide whether it is an usable breathing or cough signal
Convert the signal into a frequency-based representation (spectrogram)
Run the signal through a conveniently trained artificial neural network that would predict the user's condition and possible illness
Store features of the audio signal when the classification indicates a symptom

IMPACT

FACO will help patients get themselves tested at home, supporting in areas where tests and access to tests are limited. This will help democratize care in hard-to-reach or resource-strapped areas, and provide peace of mind so that patients will not overwhelm already stressed healthcare systems. Doctors will be able to prioritize patients with an urgent need related to their speciality, providing care from the palm of their hand, limiting their exposure and travel time.

CHALLENGES WE RAN INTO

No financial support
Working under quarantine measures
Working in different time-zones
Scarcity of high-quality data sets to train our models with
One Feature Related Problem- Legal shortcomings we might face when adding the tracking patient feature

ACCOMPLISHMENTS

We went from initial concept to a full working prototype. We got a jumpstart on organizational strategy, revenue and business plans—laying the groundwork for building partnerships with healthcare providers and pharmacies. On the creative side, we built our foundational brand and design system, and created over 40 screens to develop a fully working prototype of our digital experience. Our prototype models nearly the entire app experience—from recording respiratory sounds to reporting to managing contact, care, and prescriptions with physicians. Technologically, we successfully developed an algorithm for disease and have begun the application development process—well on our way to making this a fully functional product within the next 20 days.

You can explore the full prototype here or watch the demo (and check out our promo gif)!

WHAT WE'VE DONE SO FAR

We wanted to show that the project is feasible. Scientific literature has shown that audio data can help diagnose respiratory diseases. We provide some references below. However, it is unclear how reliable such a model would be in real situations.

For that reason, we used a publicly available annotated dataset of cough samples: It is a collection of audio files in wav format classified into four different categories.

We wrote code in Python that converts those samples into MEL spectrograms. For the time being we are not using the MEL scale, just the spectrograms. We did several kinds of pre-processing of the signals, including data augmentation, then convert all pre-processed signals, along with their categories into a databunch object that can be used for training artificial neural networks created in the fastai library. The signals within the databunch were divided into training and validation sets.

Because the dataset size was reduced, we used transfer learning. That is, we used previously trained networks as a starting point, rather than training from scratch. We treated the spectrograms as if it were images and used powerful models pre-trained to classify images from large datasets. In particular, we tried both two variants of resnet and two variants of VGG differing on their depth (number of hidden layers). This approach implied turning the sprectograms into image-like representations and normalizing them according to the statistics of the original dataset our models were trained on (imagenet). We first changed the head of the networks to one that would classify according to our categories and trained only that part of the net, freezing the rest. Later on we unfroze the rest of the net and further trained it. We finally compared the different models by the confusion matrices that we obtained from the validation test. We finally settled on a model based on VGG19. We exported the model for later use in classifying audio samples through the pre-existing interface of our mobile app.

The results are promising, especially considering the small amount of data that we have available at this moment. We have included an image of the final confusion matrix that shows how our current network can correctly classify all four categories of signal about 50% of the time, far better than the random level of 25%. We conclude that wav files obtained trough a phone mic provide information that can be useful for diagnosing respiratory condition. We are confident that we can vastly improve both the sensitivity and the specificity of our model if we can gain access to larger, more representative datasets.

We provide an image of the final confusion matrix for our model in the gallery.

This is a repository that contains the most important pieces of our work, including some code, the confusion matrix image and the exported final model.

SUMMARY

We are developing digital healthcare solutions to assist doctors and empower patients to diagnose and manage diseases. We are creating easy to use, affordable, clinically validated and regulatory cleared diagnostic tools that only require a smartphone. Our solutions are designed to be easily integrated into existing tele-health solutions and we are also working on apps to provide respiratory disease diagnosis and management directly to consumers and healthcare providers.

Feel free to click on our website for more information. We developed this website using Javascript, HTML, CSS, Figma, and integrated it with Firebase to manage hosting and our database. Thank you for reading, and don't hesitate to reach out if you have any questions!