ScribbleHero

• 

  Extinguishing the Fire!

• 

  Artistry under Hazardous Conditions.

Summary

ScribbleHero is a drawing puzzle game in which you control a magic canvas that can bring whatever you draw to life! Using 3D drawing on the snap spectacles, your task is to figure out a way to put out a burning fire using your imagination to bring your drawings to life! Your drawing is classified and evaluated using Gemini, so your puzzle solution can have infinite possibilities! So think outside of the box, get your easel ready, and dream like a ScribbleHero!

Inspiration

ScribbleHero was inspired by interactive drawing puzzle games like Scribblenauts and stories like “Harold and the Purple Crayon.” We really liked the concept of dreaming up innovative and imaginative solutions to problems, however, we wanted to actually allow the player to bring their drawings to life rather than through text. In order to solve problems that we, as developers, throw at them. In doing so, we hope to allow for dynamic environment interactions that were once thought impossible within what has been done before, and show our users what it means to dream like a ScribbleHero.

What it does

In ScribbleHero, you have a simple task: put out a raging fire that is going to burn everything down. You have a magic canvas that can bring anything you draw to life in order to put out the fire. Once you draw a solution on your canvas, hit submit, and then our Multi-modal, AI-powered (Google Gemini) input pipeline will determine what you drew, and judge whether or not it will extinguish the fire. Then, try your object out by dragging it onto the fire.

How we built it

We built ScribbleHeroes entirely within Snap Lens Studio. We leveraged surface tracking, hand tracking, volumetric lines, and Snap’s texture library alongside custom Typescript to build the program.

First, we needed to create a drawing mechanism that was both low-latency and visually pleasing. This involved multiple iterations between procedural meshes and volumetric lines before we achieved a version we were satisfied with. After this, we used a Render Target to capture the drawing of interest and feed it as input to Google Gemini.

Once the player submits their drawing for review, we used event listeners to communicate between scene objects and disable drawing mode. If the player has drawn a winning solution, their drawing becomes an object they can drag over the fire to distinguish it.

Challenges we ran into

There were a couple of primary challenges. Firstly, version control was quite difficult with git using lens studio. The gitignore did not omit some dynamic meta data that Lens Studio alters, so there were oftentimes issues in relation to corrupted files that required full rebases/repo deletions. Additionally, the Spectacles hardware limitations prevented us from fully realizing the potential of our program. This is in the context of the AI-based 3D asset generation, which takes 30+ seconds using the existing hardware.

Accomplishments that we're proud of

We finished the project! We all had great synergy and a ton of fun. We even got to meet the Scratch cat while we were here!

What we learned

The basics of Spectacles/Snap Lens Studio, dynamic asset generation, and product management/development/marketing skills, just to name a few! Shoutout to Steven, Jesse, Nick, and Alessio with the Spectacles team for helping us out!

What's next for ScribbleHero?

This program has a ton of potential to be a scalable game/ideation tool. Because Gemini handles all of the game logic, all the developers have to think of are problems to solve! We also would be really interested in integrating dynamic 3D asset generation in order to add another level of polish to the experience

Built With

• gemini
• snap-lens-studio
• spectacles