Heary Drawy

Inspiration

Picture a five-year-old who has just seen something amazing at the zoo. She knows exactly what she saw, but all that comes out is "It was big... and... it had a thing... you know, the thing!" She can't get it from her head into words yet.

Describing, listening and asking a good question when you don't understand are some of the most important skills a child builds before school, and for many kids they don't come easily. In the US, about \frac{1}{12} \approx 8.3\% of children aged 3 to 17 have a voice, speech or language disorder, and only about 55\% of them receive treatment.

Speech therapists already have a great exercise for this: the barrier game. Two people sit on either side of a screen. One describes a picture, the other draws it, using only words. But a barrier game needs an adult to set it up, run it and give feedback. Therapists are booked out, parents are busy, and most screen time asks kids to tap and watch, not to talk.

So we asked: what if kids could play a barrier game with each other, any time, and still get gentle feedback?

What it does

Heary Drawy is a cooperative talk-and-draw game for kids aged 5 and up, played on two iPads or phones.

One child is the Speaker. They see a secret picture and describe it without saying its name. The other child is the Drawer. They can't see the picture. They listen, ask questions, and draw. Both talk by holding a big button, like a walkie-talkie. When the Drawer taps Done, our mascot Drawy, a little parrot, looks at the drawing and what was said. Each child gets their own gentle feedback, and the team gets one shared score of 1 to 3 stars. The picture is revealed, the roles swap, and they play again. Kids practise describing, listening, asking good questions and teamwork. There is one team score, so nobody wins against a friend.

How we built it

A web app. It runs on any iPad or phone with no install. Every screen has big buttons, almost no reading and symbols instead of words, so a five-year-old can play without help. A Python server keeps the game fair. The Drawer's device never receives the secret picture until the reveal, so the barrier is real and not just a rule. Voices go directly from device to device. The live drawing is synced so the Speaker sees it appear and can correct the Drawer. Speech to text runs on our own laptop. An open-source speech model (Whisper) turns each clip into text, then the audio is deleted. The AI is on a short leash. Only the text and the drawing go to the AI, never voices or names. The AI only reports which features on a checklist for that picture were described and drawn. Our own server turns that into stars.

Challenges we ran into

Live voice plus live drawing across two devices, while keeping the secret off the Drawer's device. This was the hardest part to build. Voice, drawing and game state all have to stay in sync, and the one thing the Drawer must never receive is the answer. Making AI feedback feel kind. Drawy never says "wrong" and never comments on drawing skill. It praises one specific thing and asks one gentle question, and a messy drawing still gets a real compliment. Making it work when things fail. If the AI is slow or offline, the kids still get a kind neutral message and the reveal. The game never gets stuck. Children's privacy. Because the audio is children's voices, we keep it on our own laptop, delete it right after transcription, and send the AI no names. For now we test with adults only, until there is a proper privacy review for children. Real-world networks and sound. Venue Wi-Fi often stops devices from seeing each other, so we run everything over a phone hotspot. Two speakers in one room echo, so we space the devices apart.

Accomplishments that we're proud of

A complete playable round in about two days: room join, random roles, secret 3D object, live drawing, live voice, local transcription, AI feedback, reveal and role swap.
Feedback that encourages instead of judging. Each child gets feedback about their own role, the team shares one result, and the art itself is never graded.
Privacy by design. No child accounts, speech transcribed locally, no video, and only text plus the drawing sent to the AI.
A rewarding reveal every round. Kids always get to explore the real 3D object together, even when the AI is unavailable.
Small details that make it feel like a game: animated characters that hop and sparkle when picked, menu music, and motion that respects reduced-motion settings.

What we learned

Designing for a 5-year-old changes everything. Our first screens had too much text. Kids that age can't read "Hold to talk", so we redid the play screens with symbol-only buttons, big touch targets and characters instead of names. Every word we cut made the game easier to play.
The browser is a different world on an iPad. Things that worked on a laptop broke on Safari. The mic needs HTTPS even on home Wi-Fi, and the devices couldn't reach a server running inside WSL. We learned to test on the real device early instead of trusting the desktop browser.
A secret is harder to keep than it sounds. "The Drawer can't see the picture" seemed simple until we noticed how many ways it could leak: image URLs, file names, preloaded assets, even the app bundle. We had to have the server decide exactly what each player is allowed to see.
AI is more useful when you ask it a smaller question. "Give feedback on this round" got vague answers. "Did the Speaker mention the stem? Is there a stem in the drawing?" got answers we could use. We also learned to keep the score out of the model's hands entirely.
Running speech-to-text locally is very doable. We expected to need a cloud service, but open-source Whisper on a laptop transcribed each clip fast enough. It also meant kids' voices never had to leave the room.
Plan the "what if it breaks" moment first. Deciding early that every round ends with a happy reveal, even if the AI fails, took a lot of pressure off. No matter what went wrong, the kids still got their moment.

What's next for Heary Drawy

A Drawy partner for solo play, so an only child or an odd number in a class can still play. A grown-ups page where parents and teachers see which skills a child is growing, like position words or asking questions, with an at-home tip. Kids only ever see stars and kind words. Advanced mode and more pictures, with harder scenes like "a cat under a table next to a ball". Classrooms and therapy: a teacher runs a room of pairs from one tablet, and speech therapists assign rounds as homework between sessions. A pilot study with real teachers and a speech therapist before we claim any learning effect. Speech therapists invented the barrier game. We turned it into a game kids can play on their own.

Heary Drawy: say it, draw it, together.