As an experienced developer, I built some simple AR apps and realized that the power of story telling could take a leap forward if a simple-to-use tool existed for creating AR experiences. From pen/paper to the modern word processor, tools have enhanced and democratized the ability for people to create and share stories. A tool for augmented reality still needs to be invented.

What it does

ARtoo is a creative tool for building, sharing, and playing interactive augmented reality experiences.

How I built it

I built ARtoo over a weekend at an AT&T hackathon in Los Angeles. It runs on iOS and uses ARKit to provide an augmented reality experience for creating content that can be shared with others. It also uses text-to-speech and speech recognition technologies for interacting with the story. I also used Animoji characters as narrators that animate with the author's facial actions as they record audio.

Challenges I ran into

Creating a brand new tool in a field that is still evolving took some thought. There were numerous technologies that needed to be integrated in a usable manner. For the initial story experience (math challenges for children), the process of auto-scattering virtual content into the AR space was difficult. And, doing it all in a weekend was a lot of time pressure to build out the essential features.

Accomplishments that I'm proud of

The project actually works as I envisioned it, and is already a usable platform that can be built upon. It has a lot of growth potential.

What I learned

Building content creation tools a lot of work. Making complex processes simple for the user takes a lot of planning.

Current Status

Currently, ARtoo can be used to create "find it" stories. You can create narrations by recording with an Animoji character, and add questions with answers. ARtoo transforms the answers into 3D labels spread around the real world in AR space. When playing the story, the user can explore the space around them to find the right answer and tap it. I used ARtoo to create a math game story suitable for young children.

ARtoo has an audio interface that can recognize simple responses like "yes" and "no". It can also use text to speech to read story text like the questions mentioned above.

What's next for ARtoo

Building out more content options and interactive features. Adding a more robust backend for content sharing and eventually a content store.

Built With

  • arkit
  • ios
  • scenekit
  • speech-to-text
  • swift
  • text-to-speech
Share this project: