Anatomy is considered as one of the most comprehensive topics which often challenge students in general. Learning anatomy is difficult as it usually costs significant time and effort to memorize and comprehend. Our MedReality website provides an impactful solution to increase lively study environment and promote students imagination in learning anatomy. MedReality is a virtual reality website providing students exclusive experiences including several vibrant and descriptive anatomical lessons in both human and animals. Our inspiration is to allow students to both train by themselves and in groups, as multiple users can join a virtual space and experience a human and animal anatomy demonstration.
Additionally, this framework is deployed with an AI system which was trained on a cloud using human expert ground truth. The system's purpose is able to assist disease diagnosis, medical imaging segmentation or even quantification.
What it does
With the VR experience in MedReality, users may explore organs and systems in a fully immersive environment in virtual reality, display diseases in varying states of severity, and add treatments to visualize and understand medical options.
Users can exploit brains, vessels, organs, bones, and muscles in order to explore anatomical structures in orthogonal views: coronal, sagital and axial. Moreover, with a richable amount of ex-vivo and clinical scan sources, MedReality promisingly encourages users to widely elaborate their learning knowledge and skills in respect to perceive a profound understanding of structures in respect to terminology and definitions. Our MedReality WebVR is an advanced learning tool that could help not only medical students but also individuals without a medical background to visualize and explore anatomy.
In addition, MedReality also include an Artificial Intelligent which is trained using 3D deep neural network (3DUnet) to detect white matter, gray matter and cerebrospinal fluid. This AI can be further extended to detect structures in other organs like heart, kidney, liver, etc. Furthermore, it will help to correct/guide students in learning these anatomical structures. On the other hand, this AI highly aims to 1) support students and researchers working in radiology and medical imaging to segment and quantify structure volume and 2) assist medical doctors to diagnose abnormal structures in human body such as tumors.
How we built it
Our projects include different advanced technology such as Microsoft BabylonJS for VR experience, Deep Learning for training brain images and Google Cloud Platform with Compute Engine.
Challenges we ran into
- Difficult time to locate the coordinates of the mesh to import new objects onto the mesh.
- A lot of time is spent to create the contents for VR e.g. 3D mesh, scene, sounds, videos, images...
- We also had problem when trying install keras and tensorflow on Google Cloud platform.
Accomplishments that we're proud of
- Successfully use BabylonJS and implement deep learning on Google Cloud Platform.
- Learn about anatomy in higher dimension for a unique experience using VR.
What we learned
We learn to develop in:
- Google Cloud Platform
What's next for MedReality
- Develop further features and enrich the content of other anatomies
- Improve clearer and cleaner user interface -Integrate AI system to VR website- -Allow users to upload their sccans/input for viewing, segmenting and measuring.