We were inspired by Katie's 3-month hospital stay as a child when she had a difficult-to-diagnose condition. During that time, she remembers being bored and scared -- there was nothing fun to do and no one to talk to. We also looked into the larger problem and realized that 10-15% of kids in hospitals develop PTSD from their experience (not their injury) and 20-25% in ICUs develop PTSD.

What it does

The AR iOS app we created presents educational, gamification aspects to make the hospital experience more bearable for elementary-aged children. These features include:

  • An augmented reality game system with educational medical questions that pop up based on image recognition of given hospital objects. For example, if the child points the phone at an MRI machine, a basic quiz question about MRIs will pop-up.
  • If the child chooses the correct answer in these quizzes, they see a sparkly animation indicating that they earned gems. These gems go towards their total gem count.
  • Each time they earn enough gems, kids level-up. On their profile, they can see a progress bar of how many total levels they've conquered.
  • Upon leveling up, children are presented with an emotional check-in. We do sentiment analysis on their response and parents receive a text message of their child's input and an analysis of the strongest emotion portrayed in the text.
  • Kids can also view a leaderboard of gem rankings within their hospital. This social aspect helps connect kids in the hospital in a fun way as they compete to see who can earn the most gems.

How we built it

We used Xcode to make the UI-heavy screens of the app. We used Unity with Augmented Reality for the gamification and learning aspect. The iOS app (with Unity embedded) calls a Firebase Realtime Database to get the user’s progress and score as well as push new data. We also use IBM Watson to analyze the child input for sentiment and the Twilio API to send updates to the parents. The backend, which communicates with the Swift and C# code is written in Python using the Flask microframework. We deployed this Flask app using Heroku.

Accomplishments that we're proud of

We are proud of getting all the components to work together in our app, given our use of multiple APIs and development platforms. In particular, we are proud of getting our flask backend to work with each component.

What's next for HealthHunt AR

In the future, we would like to add more game features like more questions, detecting real-life objects in addition to images, and adding safety measures like GPS tracking. We would also like to add an ALERT button for if the child needs assistance. Other cool extensions include a chatbot for learning medical facts, a QR code scavenger hunt, and better social interactions. To enhance the quality of our quizzes, we would interview doctors and teachers to create the best educational content.

Share this project: