Hear, and see

• 

  Hear and see-An AR app design for educating the public of the biodiversity loss

Inspiration

I came across a YouTube video playing a bird’s chirp. The joyful song didn’t show that it was the last call of Kauai “O” o who had gone extinct already. What makes me sad is that we doomed so many species to extinction often being completely unaware of the extent of what we are doing. Here is the link of the video: https://www.youtube.com/watch?v=RLRNIrmnP2o

What it does

An app design featured Audio Augmented Reality (AAR) technology to educate the public about biodiversity loss. By identifying both extant and extinct bird species through their songs, users are expected to recognise the beautiful ones that once inhabited our planet but will never return due to human actions. AR technology enables us to experience their presence and contemplate what we can do for those who are still with us.

How I built it

(It's a design idea with an App prototype) The main feature of the app is to identify the birds' sounds and show a brief intro- -duction of each recognized bird.

What makes this app different from other bird sound identification apps is it will place a 3D AR bird that went extinct when its sound is played from the user’s headphones.

A: Record bird songs and show the bird’s introduction. It can be realized by using BirdNET’s open source for bird songs model from https://github.com/kahst/BirdNET-Analyzer.

B: Play the sound of the extinct birds and place the AR object of the corresponding bird will be placed on a given environment (trees, parks, grassland, balcony, etc.). To realize it, bird songs of the extincted birds should be trained from the above model so that the app can recognize them correctly; The ambience effect can be achieved by Unity XR audio and a headphone with AAR function. AR effect can be done by Unity XR package and Blender.

Challenges I (may) ran into

The biggest challenge can be collecting the extinct birds' sounds and training the AI model to make it recognize the corresponding bird; The second challenge is triggering AR mode to place the 3D bird by audio.

Accomplishments that I'm proud of

I feel Augmented Audio Reality is less explored than AR, which stresses more on one of our most sensitive senses but yet has been neglected in a world of visual information. This app is an idea at the current stage, but I believe it's an interesting way of combining audio with AR, especially for this specific purpose- to raise awareness of how little we know about other beings and how unaware of the extent of what we are doing.

What I learned

What's next for Hear, and see

To collect enough songs to train the model; To create 3D objects of extinct birds by referring to more pictures and introductions of them; To figure out how to integrate AAR into Unity AR development (spatial measurement, audio analyse, audio recognition, audio tracking)

Built With

• blender
• figma