Picture of human model in our AR app
Human model when the heart is selected
Data is boring, so we decided to change that. We wanted to not only use machine learning to predict a patient's risk factors based on their medical history, but we also wanted to visualize this information in a innovative way. To accomplish that goal, we decided to display this information using a mobile Augmented Reality app. This app would help doctors discover the diseases their patients were most at risk for, and it would also help patients easily understand their overall health with a 3D visualization of their body.
How we built it
We found a sample dataset of 10,000 patients from EMRbots.org with blood/urine sample data and ICD disease classifications for each patient. We picked 10 diseases affecting major organs to act as our classifications. After preprocessing the data we ran a Random Forest Regressor with SciKit learn and although we were able to find a model with low in sample error, but generalization was not very good. Our biggest challenge was that the data was insufficient to model new patients given a blood sample, meaning that in the future, we'd probably need more points as well as possibly more features that provide us with a better model.
A medical consultant provided domain knowledge to us by suggesting that many of the blood/urine samples were generally correlated to conditions concerning a few primary organs given our dataset, such as the lungs, heart, kidney, and liver.
We built the mobile app on Unity, a game creation engine, and Vuforia, an Augmented Reality SDK for mobile devices. We found open-source 3D objects to represent the human body, lungs, heart, kidney, and liver. We then created a human model using these objects, and fixed these to a point in AR using an image target. Then we developed the functionality of the app, such as fetching data from our custom API, and manipulating 3D object models.
Challenges we ran into
The first challenge we ran into was finding an adequate data set. We needed a large set of patient data to run our Machine Learning algorithm on. We eventually found a dataset with 10,000 items, containing patient data like race, gender, lab results, and diagnosis.
This was the first time we had used Unity and Vuforia for a project, so we had difficulty getting acclimated to the development environment. There were many challenges involved in getting started, like understanding the 3D environment and the material and texture of 3D objects. Once we became more familiar with the dev environment, we were able to progress at a slow and steady pace.
Accomplishments that we're proud of
We were able to build and visualize a human body and their organs on our own AR app. We took on the challenge of understanding patient lab data such as CBC and CMP data.
What we learned
We learned how to create an augmented reality app using Unity and Vuforia. We learned how to manipulate and add functionality to 3D objects in Unity using C# scripts. We also learned about the process of gathering, cleaning, and running complex machine learning models.
What's next for MLARAPP
We could improve the results of our Machine Learning Algorithm by using a much larger dataset, and including real & comprehensive patient data. We could also expand our AR model to include a more in depth representation of the human body.