Our project is inspired by Luke - One of the Personas. Here is his story:
Luke: Hi Vi, Hi my name is Luke, a Geriatrician. As you know, my job involves diagnosing health issues and plan treatment for elderly patients. Hence, I need to perform several blood tests every day to confirm the disease status for my patients everyday. My patients are in high-risked ages for COVID so I have to care for my patients virtually to ensure they are safe. With increasing concerns for COVID-19 testing and other diseases, I need to order more and more blood tests for my patients remotely via mobile blood draw service, which will send a lab tech would come to the patients’ house and draw blood.
I notice a long wait-time due the increasing demand for COVID blood tests. The laboratories such as Labcorp or Quest Diagnostics are overwhelmed with the amount of work. This long wait time can cause complications and worsen the conditions of my elderly patients. In my discussion with my virtual telehealth company and other partnered labs, we are all agree this is a big problem and really looking for a solution that can speed up this process and help reduce the wait time for patients. I see you are working a similar tech of blood counting using AI and this can potentially resolve our problem, that is why I want to talk more with you about your tech.
Vi: Hi Luke, thank you for reaching out to me! Our solution will definitely help with your problem! I am happy to go over the solution with you.
What it does
In the manual approach, a sample of blood is placed under a microscope and a pathologist manually counts the number of cells in each frame. The total count is then extrapolated by assuming that the distribution is uniform across the entire blood sample and multiplying up.
Our HemoCount platform harnesses deep learning methods to classify four types of white blood cells: Eosinophil, Lymphocyte, Monocyte and Neutrophil. Then, it counts the number of white blood cell occurrences in a blood smear
The primary advantages of this approach are two-fold:
- It requires far cheaper equipment (which will substantially reduce the cost for testing) and
- It provides rapid result on a large scale of blood cell counting almost instantly.
Below, we can see a potential time-saving effect of our AI-based platform vs the manual process:
How we built it
This dataset contains 12,500 augmented images of various types of white blood cells (JPEG) with metadata in CSV form. That includes 3,000 images divided into 4 different white blood cell types (classes). We split train-test on 80-20 ratio for each of the classes.
For this project, we used transfer learning on a pretrained Resnet34 model with a slight modification to its last layer to attain an output of 4 classes.
We predict on the testing set and print out the confusion matrix:
-Eosinophil: Precision = 0.84, Recall = 0.90
-Lymphocyte: Precision = 1 , Recall = 0.99
-Monocyte: Precision = 0.73, Recall = 1
-Neutrophil: Precision = 0.91, Recall = 0.68
Overall, the model is performing quite well on most of the cells where we have precision and recall above 0.84. However, there does seem to be some confusion (low precision rate) between neutrophils and eosinophils, also neutrophil and monocyte (low recall rate).
App development: Convert fast.ai trained image classification model to iOS app via ONNX and Apple Core ML. We develop our app in iOS because it is highly secured in privacy protection, especially for a sensitive data like lab test. This proves to be the main reason why many digital health and medical devices companies choose iOS as their main developing platform. Plus, iPad is widely used by doctors and medical staff across the world. This is the UI of our app:
What's next for HemoCount - An AI-based White Blood Cell Count Platform
In the next 2 months, our plan is:
-We will raise fund to invest more into the R&D process. We expect to implement an open-source state-of-the-art model discussed here for detecting all types of blood cells (different classes of WBC, RBC, and platelets) to improve our counting process as shown below:
-Eventually, we will expand our classes to include red blood cells and platelets. so that this platform can be widely used by the lab technologists for general blood tests. Our end goal is to make this tool so scalable that can be used in all the laboratories across the globe, even in rural areas with limited access to the internet like those in Southeast Asia or Africa.
Challenges we ran into
This hackathon project was a very different experience for us which challenged us throughout this project with the Fastai. This is the first time we all were working with Fastai and creating endpoints of the pre-trained Resnet34 model. Deploying the trained model to iOS app is also time-consuming and take us a lot of trials.
One issue when developing this iOS app is to allow users to be able to select multiple images in the app. Since Swift does not provide any built-in library for that feature, we have to use external library called DKImagePickerController. It took a lot of effort to set up the library to be able to select and retrieve images correctly. We also want to visualize the result in a bar chart which Swift also does not have built-in library for that. Thankfully, we found the open source library called Macaw which helped us build impressive charts for our app.
Working with fastai proved to be just that, fast. However, coming from a PyTorch background, understanding the distinct workflow and unique object properties of fastai was challenging, especially when the methodologies of PyTorch and fastai conflicted. Despite this, we were thankful for the rich documentation of fastai that is provided online. Aside from this, converting our PyTorch model to a Core ML model was somewhat of a process due to it being my first time experience with such tasks
Accomplishments that we're proud of
We manage to finish the project in such a limited time of 24 hours in our free time from school and work. We still keep striving to submit on time while learning and developing at the same time. We are really satisfied and proud of our final product for the hackathon.
What we learned
Through this project, we learn to implement a complicated image-recognition deep learning models from Fastai. We also learn the process of developing a mini data science project from finding dataset to training the deep learning model and finally deploy & integrate it into iOS-app. This project can’t be done without the efforts and collaboration from a team with such diverse backgrounds in technical skills.
Souza, T. (2020, April 27). COVID-19 Machine Learning-Based Rapid Diagnosis From Common Laboratory Tests. Retrieved October 17, 2020, from https://towardsdatascience.com/covid-19-machine-learning-based-rapid-diagnosis-from-common-laboratory-tests-afafa9178372