Our team wanted to build AvatarRESCUE for a number of reasons. Our strongest reason to developing this application is to mitigate human lost in disaster relief scenarios and emergency medical interventions.

We have seen natural disasters worsening every year by creating greater damage to communities and more harm to human lives. This frequency of recorded natural disasters has increased nearly five-fold in the last 50 years. Recent events that have inspired this effort have been, but not limited to, the California Fires of 2018, such as the Woolsey Fire (LA) and Camp Fire (NorCal). For scope and context, The Camp Fire burned 153,336 acres of land and destroyed 18,804 structures with most of the damage occurring within the first four hours. In April 2015, Nepal was experience a 8.1 magnitude earthquake which killed nearly 9,000 people and injured nearly 22,000 more. It is these events and stories that inspired our work with AvatarRESCUE.

What it does

AvatarRESCUE Mobile App integrates 5G Networks, IBM Watson, and Magic Leap to train non-medical persons and assist medical professional to administer intervention in Disaster Relief Scenarios in Remote and Realtime… Increase the probability of success within the golden hour. “…realtime, remote, relief…”

How we built it

First, our team created a chatbot interface leveraging IBM Watson’s Speech-to-text, Assistant, and text-to-speech integration via API calls with Unity. Second, we integrated our chatbot into a 3D Environment - leveraging Magic Leap as the interface unit - to simulate and train 2 rescue scenarios and 3 applications in CPR, fracture stabilization, bleeding out and applying tourniquet for both non-medical/medical persons to administer intervention.

Challenges we ran into

We ran into a number of challenges. Here are some from the team.

Jay Keeping time and prioritizing what our team should be focusing on within the 48 hours given to build a MVP.

Ksenia Setting up entities vs intents when using IBM Watson Assistant, that was blocking the chatbot responses. Second, when setting up nodes, there was a challenge going deep into several levels of child nodes.

John Getting models and informational panels to spatially locate in reference to a patient body reliably. Achieving interaction between android chatbot and magic leap without relying on networking stack, due to time constraints.

Joe A challenge I faced was integrating the IBM Watson API into Unity3D. As IBM is currently undergoing updates, the documentation is varied and conflicting depending on when it was written.

What we learned

Ksenia – It is easier to get intents working, however, as I have more time, I would like to work more with entities. – Building visuals for Magic Leap.


  • Focus on lowest hanging fruit to complete an MVP in a reasonably fast pace.


  • keeTime management and keepping the tasks relevant to the hackathon
  • Maintaining team mental health and wellness.
  • Worked as medical and subject matter expert relevant in the fields for the MVP

John – How to get magic leap image tracking api to function as a boolean trigger for placing objects in reference to spatially located points, rather than directly located on the tracked image itself. Learned a fast trick for making a controller object manipulation script. – For future, reference learned a high level explainer of magic leap persistent coordinate frames which enable shared anchor points between multiple devices’ maps.

Joe I learned quite a lot about Magic Leap, IBM API’s and how IBM assistant learns.

What’s next for AvatarRESCUE

0 to 1 years * AvatarRESCUE Mobile App which integrates with 5G Networks, IBM Watson, and Magic Leap One to address challenges in Emergency and Trauma Medicine in Disaster Relief (Medical Triage in Remote Areas) * Create a software platform to collect data in Emergency Medicine to develop better training models of medical triage and intervention in disaster relief * Leverage MAU as a platform for longitudinal studies and validate POC 1 to 2 years * AvatarRESCUE Mobile App provides information and communication for victims in natural disaster areas to communicate with family/friends and emergency services. * For example, Facebook Emergency Services – “Marked as Safe” feature * Create AvatarRESCUE Data Ecosystem with Health Sensors to report directly to emergency services – EMR, Paramedics, and Medical Professionals 2 to 3 years * AvatarRESCUE Ecosystem will leverage Blockchain (Ethereum) to create smart contracts under the “Samaritan Law” in which AvatarRESCUE as a system will be used to save human lives in a life/death situations – 21st-Version of Doctor without borders.

Share this project: