Sensorium

There are stories told with senses. Indeed, the most heartfelt ones are reflected other than in words. We present Sensorium, inclusive and accessible multiplayer game-interaction for sharing emotions.

Comment

Sensorium: Stories Told with Senses

Inspiration

Not all stories are told with words, some of the most heartfelt ones are expressed through our senses. Senses, that guide us and reflect our perception of the world. We were inspired by the theme fusion through the power of human emotions, the significance of embracing our feelings and play! We present multiplayer game-interaction, where all users, including differently-abled people and children, can tell their stories in a playful way.

What it does

Our AR setting game-interaction detects and shows emotions, such as enjoyment, focus, zone and even heart rate using Arctop and Magic Leap. Each of the emotions is visualized through a form, color and sound. Once users meet, they have an opportunity to choose one of the emotions from the hand menu toggled by showing their palms. Next, they throw the chosen 3D emotions into each other and forms, colors, sounds fuse resulting in a brand new combination. Additionally, users can experience particle effects on their hands that changes according to the type of emotion they choose.

Inclusivity, Accessibility, Benefits

Sensorium is inclusive and accessible. When people meet for the first time, they can tell their stories with senses, moreover, sometimes we want to communicate our feelings other than with words. Sensorium can be implemented in team buildings, between students just getting to know each other, colleagues and everyone! Differently-abled people have options to communicate and perceive their emotions through visuals and sounds. Children can use Sensorium for play and development of their senses, which is critical during their early stages of life. With bearing in mind accessibility and children we use primary colors for initial emotions which during meeting with each other fuse into complimentary colors, and of course sound speaks to our inner world!

How we built it

We first detect emotions of users using Arctop. The data is read into an android device using bluetooth and neuos SDK. The status of enjoyment, focus, zone and heart-rate are then bridged into the AR Scenes of the magic leap. The AR scenes were built using Unity, MRTK, C#. We have also integrated a multi-player interaction feature using Photon.

Challenges we ran into

There were some challenges with learning MRTK, especially for development of Magic Leap and calibration of player location for multiplayer setting. With Arctop, the primary challenge was bridging realtime data into Magic Leap that used a different os (lumin) from android. There were some challenging design decisions in materializing emotions, but we know while designing for people, we need to be them: compassion and empathy paved the way for us.

Accomplishments that we're proud of

We are proud of building a playful experience raising self-awareness, comprising inclusivity and accessibility and creating opportunities to notice precious details of life and being grateful for them. Moreover, we designed all the experience including the visualization and emotions model synching with sounds by ourselves!

What we learned

We learned to apply cutting-edge technologies with limited documentation:

  • Photon for magic leap
  • MRTK
  • Neuos SDK

What's next for Sensorium

Currently we have 9 scenarios of interactions between emotions through color, sound and form. We are intending to develop Sensorium for limitless scenarios. We are planning on making this app a multi-platform feature so it can be used between different types of devices. Additionally, we want to incorporate more players as well as complex visual user interactions. Finally, we want to create a more sustainable, scalable API that can transfer faster realtime data between Arctop and magic leap.

Built With

Share this project:

Updates