For our Metaverse health Hackathon we decided to tackle Healthcare Facility - Improving operational efficiency in healthcare facilities through the use of 3D models and immersive VR.


This past July my mom was diagnosed of cancer in her uterus. It was devastating news to me as I held the phone while the doctor talked to me about what the next steps were. I felt tears roll down my face as I couldn't control how terrified I was. I lost my dad to cancer and I just couldn't bare to lose my mom so soon. Both of us couldn't fully understand the diagnosis and a lot of the technical terms the doctor discussed were lost in translation. Fast forward a few months later, my mom recovered from her surgery and is currently receiving treatment.

I thought there should be a better way to discuss diagnosis and treatment to improve patient understanding and knowledge. This inspired us to create ApolloVR. The name comes from "the greek god of healing".

The Problem

A recent research study shows that doctors and patients are often not on the same page when discussing diagnosis and treatment. Health professionals overestimate patient’s understanding of treatment plans and diagnosis.

Communication failures resulted in



alt text

A patient has difficulty understanding all the medical terms and diagnosis given by the doctor. It’s important for us to take a step back and see how some system changes might improve communication.

Doctors agreed that better communication would help patients have more positive outcomes since these doctors could discuss diagnoses, treatments and recommendations for follow-up care.

Our Solution

We created ApolloVR using immersive VR to represent a real world environment to help doctors communicate with patients on diagnosis and treatment by innovating the way complex health information is provided and discussed.

Communication is vital to ensure everybody receives proper medical care. What doctors think they are telling hospital patients, and what those patients actually hear, may be very different. We want to provide a solution where there’s a shift in communication and there’s way to reach people in a new capacity beyond traditional brochures and office talks.

What it does

Using immersive VR to solve the communication gap between doctors and patients for effective discussion of complex health information & diagnosis. We created a Virtual Reality medical room to allow a doctor to upload 3d human body models or x-ray images (heart, eye, bone, etc) so that a patient can better visually see and understand what the diagnosis is and treatment options are.

Humans process visual data better.*

• 90% of information transmitted to the brain is visual, and visuals are processed 60,000 times faster in the brain than text.

A patient doesn't have to drive for 30 minute to an hour to access healthcare and experience wait times in the waiting room.

alt text

alt text

• A doctor can upload 3d models, x-ray depend on the patient with simple UI button a VR dashboard

alt text

Benefits of our AR/ VR Solution

• More accessible

• Less costly

• Most engaging and emotionally positive learning method

• Provide translation for all languages and easily documented

We found our solution is tackling the core issue by providing patients with visual learning tools about preventative care, and to discuss diagnosis and treatments options through immersive VR environment.

alt text

How we built it

We built it using Unreal Engine 5 incorporated NVIDIA Omniverse. We also used Adobe Premier Pro to edit 360 video compatible with UE5. We decide to use Unreal Engine instead of Unity because we also wanted to have good graphic rendering with the NVIDIA Omniverse.

Unreal 5 alt text

NVIDIA Omniverse Kit alt text

Challenges we ran into

Virtual Reality is popular but but it’s not common like using mobile phones. Not everyone has it and if you haven’t experienced you can’t really imagine what it is like to be in the VR world. We had hard time telling story about how it is feels like in the VR environment. How the doctor and patient communication works and what this VR app does.

Accomplishments that we're proud of

We are very proud of how we were able to show our concept visually. We had many users (including healthcare professional) watch our demo and everyone thought the concept was fantastic. We're also proud of learning Nvidia Omniverse & Unreal 5 through this project.

What we learned

We learned by creating ApolloVr, an immersive VR / 360 environment that simulates a real environment and engage emotions of a patient. It made complex concepts easy to comprehend. We interviewed 10 health professionals which solidified our core problem statement which helped us focus on solving the problem.

What's next for ApolloVR

We just scratched the surface of how we want to tackle this problem

  1. We want to improve the user experience for doctors and public health professionals and patients that could use ApolloVR by designing a dashboard to access patient data, 360 visual language library, and for patients to access care virtually, implement Nvidia Omniverse Audio2Face so that the communication feels more realistic.
  2. Create 360 visual language library system accessible through the ApolloVR dashboard
  3. Implement voice record documentation system so that health professionals spend less time with documentation and can spend more time communication with a patient.
  4. Lastly, we focused on the communication aspect between doctor and patient for this hackathon but we realized that this can be applied other use cases like mental health, preventative healthcare, and rehabilitation (ie. immersive 360 exercise or meditation videos etc. )

alt text

Submission Criteria:


Novelty A new way visualize health data and simplify complex health information & diagnosis. Our VR immersive solution can go beyond patient & doctor communication and can be used in healthcare space such as training, mental health space, preventative health, and more.


Built With

Share this project: