We were inspired by the Bill and Melinda Gates Foundation, as they had a challenge for medical imaging analysis in 2019 but unfortunately, we received the challenge late and missed the submission deadline. After looking at the RFP, we realized we could apply our design and healthcare skills to make a better and more vibrant solution by applying augmented reality.
What it does
ARadio is a bespoke solution offering an image-guided solution for medical scans. A radiologist/doctor can upload a medical scan and our solution will offer recommendations on the plausible medical conditions with a confidence score. Unfortunately, 2D medical scans are limited and sometimes lead to misdiagnosis, therefore we added an augmented reality component to allow doctors to view the scan in a 3D view. And using AI, we are able to extrapolate the medical problem onto the 3D models to offer inference on the growth of the medical problems, identify optimal treatment targets, print the 3D models for surgical preparations and explore possibilities of empowering telemedicine.
Our test case was chest x-ray scans for identifying pneumonia.(Normal, Bacterial pneumonia or Viral pneumonia)
How we built it
We first prototyped the solution using Adobe XD and did Usability testing to make sure the solution is user-friendly.
We then developed the web app using Angular and used TensorFlow for inference capabilities. We applied transfer learning to the ImageNet model using Keras library and open datasets for chest Xrays from kaggle. https://www.kaggle.com/paultimothymooney/chest-xray-pneumonia . For augmented reality, we used AR.js to prototype the AR viewports that work on mobile phones.
Challenges we ran into
- It was hard finding labeled open data sets for medical images
- AR glasses are still expensive, (approx $3000 per headset)
Accomplishments that we're proud of
- ARadio has won 2 awards so far since inception in April 2019. Best UX Design from AngelHack and Overall winner of IBM Call for Code Nairobi series.
- We also found a partner in Kenya to test out the solution http://www.health-e-net.org/ . They are already collecting medical image data from rural parts of Kenya.
What we learned
- You need GPU infrastructure in order to train algorithms but transfer learning made the experience a bit smoother.
- Also partnerships are key to ensure the success of this project.
What's next for ARadio
Look for additional medical image datasets especially for cancer Deploy the augmented reality component using Unreal Engine for visualizing 3D models for AR glasses.