Team SahayakAgent

Draw it on paper. Play it in your browser. Your kid's drawing becomes a game they can actually play - in their own crayon lines.

Comment

Inspiration

My nephew is three. He draws every single day. Elephants, mazes, a hero made of three circles. Many of these from cartoons or his pre-kg drawing books. The drawing goes up on the fridge and that's where it stops.

I kept thinking about the games I drew in the back of my school notebooks. Little guys jumping between platforms, spaceships, levels I designed in pencil during maths class. I never got to play a single one of them. Nobody does. You draw a world and then you close the book.

Now with advanced capabilities coming in latest AI models like GPT 5.2 - nothing is impossible. So I built the thing that would have made ten-year-old me lose his mind.

What it does

You take a photo of a drawing. Any drawing, any paper, any pen. About a minute later you're playing it.

The important part is that the game is made of the actual drawing. Not a tidied-up version, not a cartoon interpretation. The wobbly lines, the colours that went outside the shape, the character whose legs don't match. All of it, exactly as drawn, now running and jumping and getting chased by whatever else was on the page.

It reads the drawing instead of matching colours to rules. Draw spikes and they hurt, because it knows what spikes are. Paint the lava blue and it still burns. Write "3 LIVES" on the paper in your own handwriting and the game gives you three lives.

You can change the paper. Erase a wall, add a treasure, draw another monster, scan it again. The world you already have grows, and everything you already found is still yours.

Its actually => Paper, screen, paper, screen. <=

No accounts. Nothing stored, nothing posted, nothing shared unless you choose to. It runs in a browser on a phone.

How I built it

Every game is built by a chain of thirteen model calls, each one sent to a specific model for a specific job.

GPT-5.6 does the understanding. It reads the photo with vision and gives back a strict structured spec (game spec): what's the hero?, what's a platform?, what's a hazard?, where's the goal?, what did the handwriting say?.

Everything downstream is built on that spec. The hard comprehension runs on the flagship tier. The cheap repeatable checks, like screening an image for safety, run on the small fast tier, because paying flagship prices to check whether a photo contains a face is a waste of money.

Codex writes the code that makes the drawing move. For each creature it generates a real behaviour module. This elephant patrols, that one chases. Written fresh for that one drawing. Generated code never runs loose anywhere: it goes into a locked sandbox with no network, no page access, no clock, and no randomness it didn't seed itself. It gets checked, then simulated, then certified, and only then does it reach the game.

If a module fails at any point in that chain, that creature just stands still. The child never sees a crash.

The one part I feel proudest of isn't a model call at all. Before any game becomes playable, a simulator plays the whole level from start to finish, and then a real browser replays that exact route to confirm it. Every game is proven finishable before a child ever touches it. And the model's opinion can never overrule the simulator. If the model says a level is fine and the simulator can't finish it, the simulator wins.

That idea runs through the whole thing. It only claims to support what it can actually build. When a drawing asks for something the engine can't do properly, it says so instead of pretending.

Challenges I ran into

The hardest problems weren't the ones I expected.

For a long stretch, every behaviour module Codex generated was getting rejected, and the rejection message just said the id was wrong without saying what it should be. So the model guessed. For six rounds. Thirteen minutes on one drawing, and nothing to show for it. The fix wasn't better code, it was writing down the contract properly and telling the model exactly what shape the answer had to be.

Then the same thing happened with units. Nobody had written down that the game world is 960 by 540 pixels, so the model produced patrols that moved a tenth of a pixel. The elephants were technically moving and looked completely frozen. Same bug in a different costume: the contract didn't say, so the model guessed.

There was also a whole day where every generated game had dead keyboard controls in production. All the automated tests passed. It took actually opening the live site and pressing an arrow key to find it, which taught me something about what tests can and can't tell you.

The slowest problem was honesty. It's easy to build a system that always says yes. Making one that tells a child "your maze doesn't have a way through, erase a wall and scan it again" instead of quietly cutting a hole in their drawing took much longer, and it's the decision I'd defend hardest.

Accomplishments that we're proud of

The child's actual pixels end up in the game. Not a generated lookalike, not a stock sprite. The crayon stays crayon.

Ten drawings, ten playable games, every one proven finishable. Measured across repeated runs, not once on a good day.

A three-year-old's scribble works. That was the bar I set, and it clears it.

And it never lies about what it can do. When it can't build something faithfully, it says so out loud.

And more than anything - I gave this to my Sister and nephew today and that smile in his face when his scribble tuned into a game was the best part for me.

What I learned

Almost every failure came from the same place: something the system needed was true, but nobody had written it down. The id format. The units. The world size. Every time I tightened the contract instead of patching the symptom, a whole category of bugs disappeared at once.

I also learned how easy it is to build something that quietly lies to you. The early version could "succeed" by flattening a child's drawing into a generic game with their art pasted on as wallpaper. It reported success. It felt like success. It wasn't. Building the machinery that refuses to do that was worth more than any feature.

And no amount of green tests replaces opening the real thing and using it like a person would.

What's next for Team SahayakAgent

First I wanted to try integration of real world models & Meta's Animated drawing pipelines and see the difference.

Then it will be Voice. So a kid who can't spell "gravity" can just say "make the elephant my friend" and watch it happen. The plumbing is built. The interface isn't, and I'd rather ship nothing than ship it half working.

Built With

Share this project:

Updates