jen-zombie

Handheld View 2
Handheld View 1
dashboard view1
Dashboard: live telemetry panels
beach
forest
desert
snow
biomes

Inspiration

I love DOOM. Flat sprite enemies, no wasted geometry, all attitude. But it's 2026, and the screen is in my pocket now.

The thing that actually bugged me: every AR experience I've used is hand-built for one specific place. Pokémon Go needed years of curated map data. Museum AR needs a bespoke build per exhibit. That's why location-based AR doesn't scale, somebody has to author every location by hand.

So I asked a different question: what if the game just looked at wherever you were standing and built the level from that?

What it does

Point your phone at a room and the room becomes the game.

Every 30 seconds it grabs a camera frame from inside the live AR session and sends it down a two-model pipeline:

Gemini captions the scene.
Nemotron reads that caption and designs a world from it: which of ten biomes it is, how many enemies, how fast they move, how aggressive they are, and what they say to you.

The new world swaps in on the countdown with no loading screen.

Your room	What you get
A beach	Crabs and scorpions
A snowfield	Ghosts and bats
A workbench covered in electronics	A lab

Nemotron never speaks to the player. It designs the level.

You play it on a handheld I built: a Gameboy-style rig with an ESP32, an analog joystick, an attack button, and a three-LED meter.

The room reacts to how loud it is. Stay quiet and the hostile crowd walks out while peaceful creatures arrive -- calm mobs that keep their distance, never attack, and glow green. Make noise and it reverses. There's hysteresis so a cough can't flip it.

Nemotron writes dialogue for each mob based on the room it saw, and ElevenLabs speaks it aloud -- a beach zombie says beach things. The microphone is deliberately deafened while the game talks, because a zombie shouting inches from the mic would raise hostility and make more zombies shout.

You fight them with thrown slashes. Three hearts, five seconds of immunity after a hit, and a revive so a demo never dead-ends. Scoring runs walker 10, runner 15, brute 25, snapshot 50, taking a hit -20.

A live dashboard shows the whole model conversation as a thread -- the prompt, Gemini's caption, Nemotron's world -- with the actual photo each exchange was built from attached to it.

Your score settles on-chain. A read-only poller notices a run ended and mints SPL tokens on Solana devnet in a single transaction -- on death, or on demand. It never touches the game: verified by kill -9 against a live session, with frame rate, session clock and bridge uptime all unaffected. A whole run costs 5000 lamports, which is why batching works here instead of needing an off-chain ledger.

Live on devnet: https://solscan.io/token/9fn5gZCJpeyHmNc2PLxx2SR2zK9DfdadzsnvSHNt42FD?cluster=devnet

How I built it

The game runs in Chrome through WebXR + ARCore, so there's no app to install. Enemies are pixel sprites billboarded on the Y axis only, DOOM-style, so they stay upright on the real floor plane instead of tilting at you.

The network is the unglamorous half:

The ESP32 is 2.4 GHz-only and venue WiFi is 5 GHz.
My Mac has one radio, so it can't be an access point without losing the internet the models need.
A Raspberry Pi 4 solves it: it's the 2.4 GHz AP, it bridges controller state to the browser over SSE, and it proxies world requests over Ethernet to the Mac that holds the API keys.

The handheld is fully untethered -- ESP32 and LEDs running off a battery, no cable to anything. It joins the Pi's access point on its own and reconnects if it drops.

The public live-view site runs on a DigitalOcean droplet. The demo network has no internet of its own, so the Mac pushes session state out to it.

Challenges I ran into

The reasoning model was too slow to play with. Nemotron sat at a 25-second median. I benchmarked 13 configurations to find out why. Prompt-level suppression didn't work: the documented detailed thinking off convention made it worse, 1,576 output tokens versus 1,009. Only the body parameter reasoning_effort="none" actually worked. Latency tracks output tokens almost linearly.

A precedence bug that took three rounds. Photos of a rocky beach got captioned with both "rocky" and "waves," and the model coin-flipped. My first fix broke a case it wasn't aimed at: dry sand started returning water because the model generalized the bare word "beach." The rule that finally held needed an explicit carve-out, verified over 75 trials.

Bluetooth was a dead end I had to walk all the way down. The controller spoke classic Bluetooth SPP. It never established a link, and it turns out browsers can only speak BLE GATT, so even a working SPP link would have been useless. I rewrote the transport to raw TCP over WiFi.

A bug I couldn't see. The LED pin mapping was rotated three ways, so a quiet room lit yellow, a medium room lit red, and shouting lit green. The one thing the meter existed to communicate was inverted, and it looked fine.

Three timeouts in one chain, two of them too short. The Pi cut off at 20s and the game's own fetch aborted at 12s, but a successful call under API load took 24s. Both would have discarded real worlds I'd already paid for and made it look like the pipeline was broken.

Accomplishments that I'm proud of

I measured instead of guessing, and it paid off:

Metric	Result
Median latency	16× faster
p95 latency	21× faster
Biome accuracy	7/7 vs 6/7 before
Frame rate	87-94 fps
World swap	1.6 ms
Controller round-trip	6.1 ms over WiFi
Assertions	612 passing

Speed usually costs quality. Here it bought some.

Solo, in a weekend: an AR game, a custom handheld, a Raspberry Pi network, and a two-provider model pipeline.

What I learned

A fallback has to be visible as a fallback. Every failure path in this project returns something playable, but it says so, because a silent stand-in that looks like success is how you lose a day debugging the wrong thing.

The boring layer is where demos die. The models were the fun part. The thing that nearly broke it was a timeout constant.