Many people travel for work, vacation, events, or life moves. Learning about distant destinations is challenging and resources are scattered and limited.

What it does

Noah integrates knowledge about the world in a simple to navigate augmented reality experience. Combining a voice interface with highly detailed maps and informational overlays, it has never been easier to learn about the world.

How I built it

I used the Esri ArcGIS service to build a native iOS app that shows content in an ARKit view. I built speech-to-text and text-to-speech interfaces for interacting with Noah. All of the code is open-source and written in Swift 4.

Challenges I ran into

Combining text-to-speech and speech-to-text in a cohesive interface had issues with internal audio software and hardware state conflicts that lead to frequent crashes during development. The Esri data and API / SDK is extensive and a challenge to integrate without a good knowledge of all the components and tools. Fortunately, Esri staff were very helpful and available. Manipulating the augmented reality view through the Esri SDK, which is very new, had some limitations and issues that had to be worked through.

Accomplishments that I'm proud of

The app exceeded my expectations for what could be built in less than 2 days. The voice interface is very easy to use and allows for a seamless interaction with the augmented reality visuals. The map interaction experience is inviting and addictive, engaging the user to explore and learn.

What I learned

The speech input / output coordination on iOS is very tricky to get right, but I found a balance that worked well for Noah. The map content, visuals, and integrated data becomes a very rich experience when presented in the right way.

What's next for Noah

Integration with more data sets for the map. Addition of more destinations. The ability to add commentary to the map experience and share it with others such as family or travel partners.

Built With

Share this project: