Augmented Reality app for sound-based navigation

Team ARound

Anandana Kapur, Dhara Bhavsar, Maxym Nesmashny, Sunish Gupta, Vedant Saran

User types

Visually impaired (sound reliant)

Visually and hearing impaired (haptic feedback reliant)

Mainstream (information and context reliant)

Why this app?

Team *Getting ARound * wants to augment the world with sound. Places of interest emit sounds that alert you of their presence, and that adds another layer of information to the world around us.

Spatial audio and ASMR related sounds can help to augment our everyday experiences and be a very powerful tool to navigate the world.

Contextual audio cues that do not interfere with existing sounds or cause disorientation can be very valuable to navigate cities, forests, etc.

What does ARound do?

The Augmented Reality app ARound equips all its users to independently navigate spaces by augmenting sounds of places, objects, and activities. In its current state the app functions using three modes:

  1. Discovery mode: As you walk down a path, we use GPS data with combination of ARCore camera technology to sense your surroundings and trigger sounds of objects and locations on your path depending on your orientation towards them. The sounds are geofenced to an area so that they start playing just before you approach an object/place of interest, and gain in volume as you get closer.

  2. Breadcrumb mode: As you walk around, the app leaves a trail of sound along the path you have taken. If you lose your way or need to retrace your steps, you can simply follow your "audio breadcrumbs" to follow the path you took. Since the sound is anchored to the real-world, and we can accurately track the user’s position using ARCore, the sound is spatialized to the user’s orientation relative to the breadcrumbs. That makes it super easy to get back on the path - you just follow the direction the sound appears to be coming from. This feature gives the visually impaired the freedom and confidence to go on walks, hikes, or boating, without assistance.

  3. Share mode: You can choose sounds from a library of audio files and link them to locations in the real world, through a web application. This creates a sound pin on the map that is triggered when the user approaches that location in the real world. This could be used to set up meetings with a blind person - You’d just drop a pin of an audio note at the meeting location, and they’d know when they arrived. There’s also a social media aspect to this- you could create and share your own ‘sonic maps’ of your surroundings to experience the unique sonic landscapes that matter to your friends and family.

Why is ARound unique?

ARound brings a new perspective to the possibilities of Augmented Reality. While most of AR today is centered on augmenting one's visual systems, we think that there's immense unrealized potential in augmented one’s sense of hearing. While current-generation AR systems can only augment your visuals in a tiny field of view, spatial audio enhances the user’s perception in all 360 degrees.

It takes advantage of the spatialization and contextualization abilities of AR technologies to create a highly responsive and immersive experience. With good computer vision and enough sound pins placed on the map, ARound could build a vibrant ‘Internet of Sounds’ - an immersive and contextual layer of sound augmented on top of the real world.

Current features

  1. Learn about their environment

As the users walk through the world, they’ll become subtly aware of all the sounds and tagged objects around them. That’s important for the blind - objects that are quiet are invisible, so augmented sounds help you orient yourself and learn about objects in your surroundings. Haptic feedback could make this useful for the deaf as well. That’s also interesting to mainstream audiences as a discovery or educational platform.

2. Path-finding and **_Deviation detection _**

Easily find your way back on hikes and new paths. Hands-free and don’t need to be looking at the screen, so you can navigate yourself which engaging with the real world. If you deviate too much from your path, the app will gently alert you of it, and guide you back in the correct direction. Since the deviation alert is volume-based, you always know exactly how far away you are from your path.

3. **Social media**

Sonic scapes can be created and shared by all. Users can create sound maps for fellow travelers or teachers can make students discover cities and events.

*What inspired us *

Independent living and traveling by people with disabilities have always been a challenge. This is especially true for the visually impaired population. Activities such as walking around or getting to familiar places like cafes, post office, ATM etc. can be challenging. Getting around unfamiliar routes while hiking, sailing or on nature walks can be be even more challenging.

Challenges we ran into

Designing for app for the blind presents a whole new set of design challenges, many of which we were initially unaware of. Sunish’s experience in accessibility design was crucial in designing a product that would actually be usable by the target audience. However, we learned that the Unity engine was not natively compatible with Google Talkback and the other accessibility features on Android. The plugin to enable talkback was a paid feature, and under the licensing rules of the hackathon, using it would’ve disqualified us.

We also found out that our phones were not supported by the ARCore platform. Thanks to the ar-for-all guide, and the help of the mentors at the event, we finally managed to get it to work on (some of) our phones.

*Accomplishments *

We are thrilled to have built a working prototype. We were able to implement directional spatial audio that leads you along a certain path. We are proud about the idea being user-validated by our team member who is visually impaired.

What we learned

None of us had a lot of prior experience with sound design or spatial audio, so we learned a lot about that. We got to play with ARCore and Firebase. Some of us improved our understanding of how to work with Unity better. We brainstormed about translating user experience and feedback into technical solutions and that was extremely rewarding. We even learned how to incorporate accessibility/inclusivity design in our product. Our collaboration skills improved as well!

What's next for ARound

We are looking to develop our MVP into a full-scale platform that would help all our user types* to navigate a given environment as well as be fun experience to use in everyday life. Finally, we want to build a full-scale platform for augmented sound, something like ARcore or ARkit, but with spatial direction in mind. With the help of this platform, many developers would be able to jump in and build all kinds of amazing apps and experiences. We are very excited for the future of ARound.

Share this project: