Portal Reef

• 

  This is a concept art of Portal Reef, UX & UI design were developed from this sketch.

• 

  Render image of 3d builds using Blender & Unity

• 

  Shows handtracking in Hololens 2. Users will be able to interact with corals using hand tracking.

• 

  Game development in Unity

• 

  Shows design of informative bubble above specific endangered coral reef species for users to learn.

• 

  Illustrates coral reef destruction at the Great Barrier Reef, comparing years of 1999 & 2020.

Inspiration

People often talk about what you should do to make change for climate change. New technology brings people together to connect, impact, and change. Providing a powerful interactive experience with endangered coral reefs through Hololens 2, we hope people will remember the feeling from human touch and coral reef destruction and experience why it's important to take action for preservation.

What it does

Portal Reef is a multimodal sensory project with an interactive immersive environment shown through Hololens 2 and Neuos and EEG-based interface SDK to raise awareness of coral reef destruction or coral bleaching. We're using visual, auditory, and hand tracking to create an interactive environment for users to experience how human touches trigger coral reef destruction. The audience will be running into fish, seashells, and endangered coral species in the scenery. When they approach the endangered corals, the informative bubble will be available for people to learn about the characteristics of each endangered coral.

How we built it

• Rina, the UX&UI Designer pitched the idea of coral reef destruction and interaction. The idea was developed throughout the week, and she created the concept art of the underwater environment. From there Rina developed the production design of the scene, where models would be placed, how people will interact with models, and in what way. Bringing multi-sensory using visual, auditory, and hand tracking she developed a user interface with contrasting colors and music for the environment.
• John, the 3D Artist researched free 3D models and imported them into Blender and modified them, and arranged them in a scene for renders. He rendered and exported the models as fbx files for Unity.
• Andre, the developer imported the models into Unity and placed them within the scene according to the storyboard vision. Unity (C#) scripts were then created to have the models move within the environment. For example, fish objects have a script that directs them to revolve around a target location with an assigned speed. To create a more immersive experience, Andre leveraged MRTK's Scene Understanding features which identify and segments a room (i.e., floor, walls, ceiling) which he then renders a video texture over the sectioned piece. In this case, we used a video of an ocean to simulate the projected view of an ocean surface from below onto the ceiling. The info panels that give informational facts regarding the endangered coral species use MRTK's Orbital solvers to rotate the panels until it faces the users at all times making it easier for the user to navigate through the virtual environment. We imported music developed by Rina to add to the emersive experience which plays on Awake.
• Hairuo, the developer connected the Arctop headband with its application (Neuos) over Neuos SDK to read brainwave data. This brainwave data shows the level of enjoyment a person is having. Then, we sent the brainwave data through a self-defined Android application to an online database called "Firebase". Then, we download the data from firebase directly down to a C# script and combine it in Unity.

Challenges we ran into

Set up Nueos Android Appllication on a windows PC, which wasn’t supported. We then borrowed an Android phone eventually to make it work. Send data from an Android App to the firestore database. Download data into a C# code in Unity.

Accomplishments that we're proud of

Get data from Nueos Brainwave headband to a database.

What we learned

How to install a windows subsystem for Android Application. How to collaborate across different teams to resolve issues. Learned to use Nueos App and use firebase.

What's next for

Continue developing and exploring ideas in XR to encourage people to provide better care and attention to Coral Reef Destruction and Coral Bleaching. Explore other brainwave data applications in XR. Also, continue researching ocean ecosystems and add more endangered species, completing a dynamic underwater environment. More interaction and connection with organizations such as Greenpeace, NOAA Fisheries, EDGE, WWF, and Reef Resilience Network.

Built With

• arctop
• arctopneuos
• blender
• c#
• c++
• hololens2
• mrtk
• unity