http://print.drawmaticar.com

Inspiration

What if you can instantly turn into AR whatever you draw on paper.

What it does

Lets you write on real paper to create an 3D AR story (as read and interpreted by AI) that is also read by a neural voice

DrawmaticAR is an AR magic drawing app for those of us who can't draw. It's an AI visual storyteller that lets you turn your words auto-magically into AR.

How to use it

Hand write your story on real paper (officially, known as the DrawmaticAR MarkerPaper).
When you're done, open the DrawmaticAR app on your iOS, Android, or Windows Holographic device and point it over the MarkerPaper. Press the big button that replaces the middle rectangle.

Using the Marker Paper, we're sending just the perspective re-scaled crop of what you wrote to Azure! (It actually does not matter what angle you take the photo at, as long as you can see both the box you've written in and the button is tracking well.)

The following Microsoft Azure AI solutions will work AI magic: Computer Vision recognizes text that TTS reads with a neural voice and Text Analytics identifies salient keywords from landmarks like the Eiffel Tower to oddities such as dancing pirates.

Before you know it, your words have turned into a drawing in 3D, as it pops into AR - and your story is now told by AI!

Please note the currently limited lexicon of 212 tag words: http://drawmaticar.com/lexicon

How I built it

Azure Computer Vision
Azure Text Analytics API
Azure Text to Speech API with neural voice
RealityScript C#
LAMP
Vuforia
Unity iOS/Android/Hololens (Windows Holographic)
AR Marker designed in Photoshop/Illustrator

Font: Renaissance

Challenges I ran into

At first I tried drawing flow charts, but realized that that wasn't supported. I played around with what worked with Azure Computer Vision and found that the hand writing API is quite robust! Text Analytics is able to pick up salient nouns to load objects from the 3D database and then neural TTS voice reads it wel!

Accomplishments that I'm proud of

Last minute ideas made somewhat coherent!

What I learned

I really like the elegant design of Azure REST APIs, on the part for not having to install a bulky library to access a simple REST based API, and was glad that I was able to use PHP curl to use the Text Analytics API. In contrast, both Google Cloud and AWS require complex authentication and I/O methods that make not using their library a total pain.

Microsoft's TTS solution offers custom voice creation from your own voice! Seems to require recording a lot of footage... If I had more time...

What's next for DrawmaticAR

positioning of the AR object close to the word on the page - the bounding box is provided from the API, so this was something I had planned to implement before deadline but had to sacrifice to keep polish.
more animations
add more adjectives modifiers for nouns (currently just dancing pirate)
add more than 212 keywords to search
add my own voice or other voice opptions to the reading