Inspiration

We wanted to explore what it would look like if AI could interact with something you create in the real world instead of just responding inside a chat box.

That led us to a simple idea: what if making a game level could start with nothing more than a sheet of paper and a few markers?

Sketchquest lets you draw a platformer level by hand, take a photo, and turn that sketch into something you can actually play.

What it does

Players draw their own level using a simple colour system for platforms, hazards, coins, enemies, the starting point, and the goal. They then take a photo or upload the drawing.

Gemini analyzes the image and converts the sketch into structured level data. Before the level reaches the player, Sketchquest checks whether it is actually possible to complete. If a jump or layout is impossible, the system can repair the level before loading it into the game.

The result is a playable Phaser platformer based directly on the player's drawing.

During gameplay, the experience continues to react to the player through events such as deaths and wins, including a narrator that can comment on how the player is doing.

How we built it

The frontend is built with React, TypeScript, Vite, and Phaser 3.

When a user submits a drawing, the image is sent to our Node and Express backend, where Gemini Vision interprets the sketch and returns structured JSON representing the level.

That level then goes through a sanitization and reachability pipeline. We use the same game physics assumptions as the Phaser game to determine whether the player can reach the goal. If the level is not playable, Gemini can receive the validation result and attempt a minimal repair.

Once validated, the level data is sent to Phaser and converted into platforms, hazards, coins, enemies, a player spawn point, and a goal.

We also built mock, caching, fallback, and record/replay systems so we could continue developing and demonstrating the project without depending on constant Gemini API access.

Challenges we ran into

One of the biggest challenges was turning an imperfect real-world drawing into reliable game geometry. Photos can contain shadows, perspective differences, inconsistent marker strokes, and drawings that do not line up perfectly with game physics.

Another challenge was making sure AI-generated levels were actually playable. A level can look correct visually while containing a jump the player can never make, so we built a separate reachability validator and repair loop rather than relying entirely on the model.

We also had to account for Gemini latency and API limits. To keep the project reliable, we added local caching, mock responses, daily request limits, recorded fixtures, and fallback levels.

Mobile capture introduced another challenge because direct camera access requires HTTPS, so we also had to make sure the frontend could work reliably across both laptops and phones.

What we learned

The biggest thing we learned was that using AI effectively is not just about sending something to a model and displaying the response.

Gemini handles the interpretation and reasoning, but the rest of the application still needs traditional validation, error handling, game logic, and fallback systems around it.

We also learned how useful structured AI output can be. Instead of asking Gemini for a description of a drawing, we use it to produce data that the rest of our application can immediately act on.

What's next for Sketchquest

One direction we would like to explore is multi-page levels, where players could draw several sections of a larger world and connect them together.

We would also like to expand the editing experience so players can modify their generated level using natural language and direct interaction rather than redrawing the entire page.

Ultimately, Sketchquest is about making digital creation feel more immediate: draw an idea in the real world, show it to the computer, and start playing it.

Built With

Submitted to

Emberhacks Day 2026
- Winner 1st Place

Created by

I worked on the Gemini pipeline that turns a hand-drawn sketch into a playable level, from the vision call to a custom solver that simulates the player's real physics to make sure every generated level is actually beatable, with an auto-repair step that fixes broken ones without extra API calls. I also built the core gameplay in Phaser, the hand-drawn art style, and the roast narrator. Watching a photo of markers on paper turn into a real, playable platformer in seconds was the moment that stuck with me. It genuinely expanded what I thought was possible to build in a day, and got me a lot more excited about how far you can push an API when you treat it as one piece of a bigger system instead of the whole thing.

Vansh Parikh
shayan irfan
Mihir Mistry