• 

  Logo

Grow A Game

Say it. Play it. Together.

Inspiration

We all remember what it felt like to play with someone. Passing a controller on the couch, arguing over the rules of a game you made up, laughing at something that only made sense to the two of you. It's one of the easiest ways people connect, and most of us never think about it.

A hospital bed takes that away. In the US, children have about 5.2 million hospital stays a year, and even setting aside newborns, that's roughly 1.5 million stays for kids pulled out of school, away from friends, and stuck in a room that isn't theirs (AHRQ HCUP). At the other end of life, 1.72 million Medicare beneficiaries were enrolled in hospice in 2022, and 49.1% of Medicare decedents that year were in hospice care when they died (NHPCO). The time is short: half of hospice patients receive care for 18 days or less (NHPCO).

In both places, the people who love the patient show up wanting to connect, and then don't know what to do. Visits turn into sitting quietly beside the bed, watching whatever is on TV, or scrolling on separate phones. Games should help, but they're built for a different person: someone healthy, with steady hands, plenty of energy, and time to learn the controls. They need shared controllers passed from hand to hand, which is a real problem in a room where infection control matters. And every game is the same generic content for everyone.

The patient is also almost never the one making decisions. Doctors decide the schedule, nurses decide the meds, families decide the visits. We kept coming back to one question: what if the patient got to create something, and everyone else got to join in?

That's what we built.

What it does

Grow A Game is a laptop console and phone-controller prototype that turns anything you can say into a game you can play together, built around three ideas: create, join, connect.

🎙️ Ask. On the record screen, the patient holds Record and describes the game they want in their own words, like "a dragon who collects stars in outer space" or "a racing game where my dog is the driver." The laptop microphone records it. Grok transcribes it, and the words appear on screen so they can see the console understood them before the game is built. There is no typing and no menu of existing games.

🧠 Create. Grok builds a playable game from that description and opens it on the laptop screen. A named game, such as chess or Connect Four, is built as that game. A personal or made-up idea becomes a small original game about the people and places in the sentence. While it works, the screen says what is happening: listening, finding or drawing art, writing the game, testing it, then sending the controllers. The patient can cancel and go back to the record screen. They aren't just a player. They're the game designer, and the game is theirs.

🎮 Adapt. Each game brings its own phone layout, chosen for that game: a D-pad, a joystick, buttons, or a grid of labeled choices such as Connect Four columns or battle moves. The layout can change in the middle of a game, so walking and choosing a move don't have to share one control scheme. One-player games open one phone. Two-player games open two.

📚 Keep. A signed-in player gets a library on the record screen. Each saved game uses the title Grok gave it, so yesterday's game can be opened again with today's visitors. Someone who continues as a guest sees a warning first: guest games are not saved, and they will not appear in a library if they leave.

💛 Connect. A grandparent in hospice can describe a game from a story in their own life and play it with the people in the room. A child can ask for a game that has never been made and have a friend join on a second phone. Each person uses their own phone, so the controller does not have to be passed from hand to hand.

How we built it

Hardware. The console is a laptop. Its screen is the game display, its microphone takes the recording, and it runs the server. Players use Android phones plugged in over USB. When a game is ready, the laptop opens the controller on each phone: the first phone is Player 1, and the second is Player 2 when the game needs two people. The controller page is laid out for a phone held sideways.

Console shell. A full-screen browser app is the launcher. It opens on a CRT-style home screen, then sign-in (or Continue as guest), then the record screen, "Let us build your story." Generating a game switches to a black loading screen with a live status line and a cancel button. The finished game opens as its own page, with a way back to the record screen.

Speech pipeline. Record starts and stops a recording from the laptop microphone. The audio is sent to Grok speech-to-text, and the transcript is shown in the text box before anyone presses build.

Game generation. The transcript goes to Grok with a system prompt that enforces a strict JSON contract. The model returns a short title, whether the game is for 1 or 2 players, a controller layout for each player, a JavaScript game built on our runtime kit, and scenario tests. Games are drawn on a 2D canvas in a chunky Nintendo DS pixel-art style. Three.js is available when a game actually needs 3D. Named games reuse public sprites we already know how to draw, such as chess pieces, Pokémon, and playing cards. Original ideas get newly drawn pixel art, or simple emoji sprites, taken from the sentence the patient said.

Validation. Before a game is marked ready, the server checks the code: it has to use our kit, it can only import that kit and Three.js, it has to include real scenario tests, and a two-player game has to read both players. A headless browser then boots the game, drives the controllers, and runs those tests. If the check fails, the code and the failures go back to Grok for a fix. If the game cannot even start, it is rewritten once.

Dynamic phone controller. Each phone loads our controller page, which reads that game's controller.json and builds the layout from a small library we wrote: D-pad, joystick, button, and choice. The game can swap a player's layout mid-round over the same WebSocket that carries button presses. Input on the phone shows up in the game immediately.

Wired connection. The phones do not need the room's Wi-Fi. Android's adb reverse makes localhost on the phone point at the laptop over the USB cable, and the laptop opens the controller URL in the phone's browser. The same controller page can also be opened from another browser on the laptop's local network.

Self-repair. The game runtime catches script errors and failed promises, including the file and line number. It sends that report to the server, which sends the error and the game's code back to Grok, saves the fix as a new version, and the game page reloads. Each game gets up to three of these repairs. If a repair cannot be made, the screen says so instead of failing silently.

Challenges we ran into

Getting the AI to follow the rules. Generated games kept importing things they were not allowed to use, skipping the tests, or building a two-player game that ignored the second controller. A strict JSON contract, static checks, and a headless playtest before the game is shown were the difference between a demo that works sometimes and one that works every time.

Talking to Android phones over a cable. Plugging in a phone does not make it a controller. We use adb to forward a port over USB and open the controller page on the phone. The first version of that command was split by the phone's shell at the & in the URL, so player=2 was dropped and both phones opened as Player 1. Quoting the URL fixed it. We also lock the controller UI to landscape, because a portrait page is a bad fit for a D-pad and buttons side by side.

Making the wait feel like part of the game. Looking up sprites, drawing art, writing the game, and playtesting it all take real time, and a blank spinner kills the mood in a room. The loading screen names the current stage, and cancel returns to the record screen.

Designing controls the model might ask for. Since each game picks its own controls, the phone had to handle any combination from a fixed set. D-pad, joystick, button, and a grid of choices cover board games, arcade games, and battle menus, and the game can switch layouts when the moment changes.

Tone. This project touches kids who are sick and people at the end of their lives. The home screen, the record page, and the guest warning had to feel warm and playful, never clinical and never sad.

Material Constraints. We started with grand ambitions on what the full machine would look like. Unfortunately, time and material constraints ran in the way for this prototype.

Accomplishments that we're proud of

• A real end-to-end loop: spoken idea, transcript on screen, generated game, playtested, running on the laptop, with controllers opened on USB Android phones.
• A controller that rebuilds itself for every game, on phones plugged into the laptop.
• A wired path that does not depend on the building's Wi-Fi.
• Games that send their own crashes back to the model and reload the fix, up to three times.
• Original pixel art for made-up games, and real public sprites when a known game already has them.
• A library of games named after the games themselves, and an honest guest mode when someone does not want an account.
• A tool where the patient creates instead of just being cared for.

What we learned

A game is a reason to be together. The social part of the design mattered more than any single feature. Nobody remembers the graphics. They remember who they played with.

Generative AI needs guardrails to be useful. The hard part was never the model call. It was the contract, the playtest, and the repair loop that make the output reliable enough to hand to someone in the room.

Designing for a real place changes the build. Separate phones, simple generated controls, a laptop that can run on its own battery, and a cable that does not need hospital Wi-Fi all came from thinking about a hospital room instead of a living room.

Hardware and software fail differently. A USB command that drops half the URL and a bad prompt can look identical from the outside, so we learned to test the phone link and the game on their own before blaming the other.

The controller can be part of the game design. Because the layout is generated with the game, a round can start on a D-pad and switch to a list of choices when the rules change, instead of asking every player to learn one permanent control scheme.

What's next for Grow A Game

• Pilot with Child Life specialists. Hospitals already use play as part of care, and we'd want to test with the people who know these patients best.
• Remote play. Let family members who can't visit join from anywhere, so a parent working late or a grandparent across the country can still play.
• Accessibility modes. Eye-gaze, switch, and single-button controllers for patients with very limited movement.
• A change in their own words. Let the patient ask for a change to the game in front of them, and rebuild that game instead of starting over.
• Memory games from photos. Spoken memories can already become a small original game. Next is letting hospice patients turn family photos into games their families can keep.
• A dedicated console. Replace the laptop with a small single-board computer and screen, in a sealed, easy-to-clean body a hospital could own, with phone docks built into it.

Built With

• grok
• three.js