- Whenever I have a pain in the chest, leg or arm, I never know what to think or what to look up. It would be quite unnecessary to look up leg pain, that is so vague. And I am not able to name each part of my leg. There had to be a better way to asses that. After all, there are only 2 in 3,000 people that are trained medical professionals. What if we could enable anyone to determine what their problem is.
What it does
- Our iOS app provides an Augmented Reality experience backed by a Computer Vision algorithm to assess your symptoms when you are feeling ill and provides you with the most probable diagnosis. If you are feeling sick or having some sort of pain, a person would place pinpoints on those areas of pain on your body. The app would then process those pinpoints to provide you with a list of possible issues. AR here allows you to be extremely precise with indicating what area of your body is hurting/uncomfortable.
How I built it
- We built our app using ARKit and Swift. Our API is built in NodeJS and hosted on GCP. Our Machine Learning algorithms used Caffe and OpenCV for Computer Vision. Our website is written In Vue.js and also hosted on GCP. The website is live as well.
Challenges I ran into
- We had a ton of issues with everything from domain deployment to post request issues.
- Figuring out the best way to translate 3-dimensional nodes in ARKit to usable coordinates for the ML algorithm to figure out the exact body part the node points to.
Accomplishments that I'm proud of
- iOS app is working, API is live and the website is almost done.
What I learned
- Learned about SceneKit which could be used for making iOS games and about ARKit which is for Augmented Reality.
- We learned a lot regarding API calls and how different technologies integrate and work together.
What's next for ExaminAR
- Better visualization of the AR, using for example an overlay of the anatomy. We did not consider this idea because of the cost of those anatomy models.
- Ability to use a front facing camera and thus not require assistance to operate.