Art-do-be

Inspiration

When I was little, I drew all the time: dogs with too many legs, houses with the sun sitting on the roof, a whole zoo on the back of a school worksheet. They went up on the fridge, then under the next drawing, then somewhere I'll never find again. I still think about them. I'd give a lot to walk back into that zoo and see the dog I drew.

That's where Artdobe comes from. A kid is proud of a drawing for about a minute, and then it's gone. We wanted to keep it for them. Draw a cat and it walks off the page. Tap it and it says hello. And the whole planet is made of what kids drew, so years later you can still go back and find your dog.

What it does

Draw anything, or upload a picture. Gemini looks at it and finds what it is: an elephant, an owl, a house, a fish. Then it comes to life on the planet right away. The elephant walks, the owl flies to the sky, the fish dives into the nearest pond.

Tap the dog and talk to it. Gemini writes what it says, and ElevenLabs gives it a voice (a duck quacks first, then says hi). There's quiet music if you want it.

Then it's your choice: hang your paintings in frames on the Toy Globe, or let them live on the Paper Globe.

Safe by design

A kid's planet is private by default. There's no chat with strangers, no ads and no feeds, and a kid can only talk to their own drawings. Drawings stay on the device. Only a small image for Gemini and a reply for ElevenLabs ever leave it. Characters keep to one short, kind sentence. To scale, we'd add private accounts owned by a parent, so a planet can follow a child across devices and stay theirs.

How we built it

No 3D models, no audio files, no engine. Everything is generated in the browser.

1. Cutting the drawing out. On the whiteboard, the strokes are the drawing, so we don't need to segment anything. We close the gaps in the outline with a dilation, which uses an exact Euclidean distance transform, then flood-fill from the border. Whatever the fill can't reach is the shape. A second distance pass gives the black outline and the white paper border. Marching squares traces the edge, and Douglas–Peucker simplifies it to a clean polygon. It takes about 150 ms.

2. Knowing what it is. Gemini gets a small JPEG and returns JSON: a kind from our table, a name ("Strong Tomato") and which way it faces. One table (kinds.ts) maps each kind to its shape and its movement (walker2, walker4, bird, insect, fish, floater, static). That answer is the behavior, so a new creature is one line.

3. Living on a sphere. Everything moves in the tangent plane at its own spot on the planet. Each step goes forward along the surface, then we re-project the heading onto the new tangent plane so nothing drifts or flips at the poles. Orientation is a quaternion built from the surface normal and the heading.

4. A planet of paper. About 900 cut-outs share one vertex shader that bends and sways each sheet. They are instanced, so it costs one draw call per cut-out kind.

5. Voice. Gemini writes one sentence of 12 words or fewer. ElevenLabs does the rest: sound effects for animal noises (made once, then cached), text-to-speech with four voices, and speech-to-text for the mic. A character's voice comes from its name, then its kind, then a stable hash of its id, so it always sounds the same.

Stack

Vite + React 19 + TypeScript: the app and the typed contracts between parts.
three.js + react-three-fiber: the planet, the instancing and the paper shader.
Zustand: world and player state, read inside the render loop without re-rendering React.
framer-motion + Tailwind: the UI around the world.
Gemini API: recognition and dialogue.
ElevenLabs: voices, animal sounds and transcription.

Challenges we ran into

Scribble to behavior. A child's drawing has no labels and rarely a closed outline. The cut-out had to survive gaps and smudges, and whatever Gemini said had to turn into a way of moving: a cat walks, a bird flies.

Calling Gemini and ElevenLabs on time. Kids won't wait. Models come back 503 (busy), 429 (quota) or 404 (not on this key)

What we learned

A simple product can still give people a lot of power. A kid draws one thing and gets a living, talking character back, and the app handles everything in between. We cut buttons every time we pictured a six-year-old using it.
We kept the rules plain. Gemini picks from a short list, and the list decides how each thing moves. We kept the stack small and utilized as much of Gemini and ElevenLabs's power as we could.

What's next

A thin server in front of Gemini and ElevenLabs, plus parent-owned private accounts. Right now the keys sit in the client and drawings live in one browser. That works for a demo but not at scale.