Magic Brush XR

Inspiration

We were interested in capturing the magic of childhood experiences such as painting and reading picture books. We wanted to leverage nostalgia and child-like wonder to facilitate learning. We started considering how we could use fairy tales to promote learning, and that was where we realized we could use a fairy tale to teach a foreign language. We then determined that it would be most effective if we chose a story whose native language we wanted to teach. We chose the folktale, "The Magic Brush," from ancient China because the metaphor of the paint brush bringing things to life is a perfect metaphor for Spectacles and augmented reality.

What it does

Our experience teaches guests Chinese in a playful narrative experience where the guest paints with physical paints, and the Spectacles application recognizes what they paint and integrates the painted objects into the story. The experience provides the translation for painted objects, helping guests learn the Chinese words for key story elements.

How we built it

We developed the experience in Lens Studio. We used meshy to generate several of the 3D models, and we touched up the models in Blender. We used Roboflow and Paper Space for training the AI models we used in the experience.

Challenges we ran into

Optimization was a big challenge, as many of the 3D models we generated had large poly counts, and augmented reality experiences require frugalness when it comes to poly count and model sizes. We had to invest some time in optimizing the models, simplifying animations, and in some cases, converting 3D models into images in order to save space.

Accomplishments that we're proud of

Our team was excited to tackle training a custom ML model! We needed to track painted real world objects accurately and reliably in order for our experience to work, so a simple AI agent wouldn't be reliable enough. We used Roboflow and Paperspace to generate training images for the custom ML model. We experimented to find reliable objects to use for the training.

We are incredibly proud of what we created. We are proud that we managed to create a complete narrative experience that tells a story from beginning to end. And we are proud that we were able to realize such a novel concept!

What we learned

We learned that it's important to start building in-engine as quickly as possible, as pressing needs such as the need to optimize or reduce rendering demand become apparent earlier, allowing for more time to recalibrate, adjust our approach, and optimize, as needed. We also learned about texture sequencing as an incredibly useful optimization method.

What's next for Magic Brush XR

We are very proud of what we made, and we would love to add additional features and content in the future. There were several features that we wanted to add, such as Chinese character writing recognition and practice, additional stories and vocabulary words, and color detection capabilities.

Built With

• lens-studio
• paper-space
• roboflow
• spectacles