SnailMail

• 

  Opening scene

• 

  architecture

Inspiration

My grandfather taught me to solve a Rubik's cube. I loved that puzzle. Since then, it’s mostly sat on a shelf, waiting to be scrambled and solved again. I wanted something more, A way to turn that familiar cube into a higher-level challenge.

Around the same time, I’ve been fascinated by machines that see the world: self-driving cars, robots, medical devices navigating chaos. I wondered if a simple system could understand just a little of my messy real world, without the need for giant, brittle models.

I also love video games, but NPCs always speak in canned lines. What if dialogue could be dynamic, guided by a story but alive with surprise?

And then there’s the cube itself. I can solve it, slowly, but I wanted it to do more... something fun, something different.

Reading Terry Pratchett’s Going Postal added the final spark. Working in a postal service, with its chaos and whimsy, felt like the perfect stage for these ideas to collide.

All of this came together this project: a game called Snail Mail.

What it does

You’re a snail with a lifelong dream: an ill-advised holiday to Salt Lake City. To afford it, you must deliver packages between towns. But the path you follow isn’t a little weird... it's built from a real-world Rubik’s cube. Place a red tile, and fire leaps from the ground. Place a blue tile, and the flames die out. As you travel, mischievous enemies appear, forcing you to dodge, outwit, or face them in combat.

How I built it

The game runs in your browser, built with Phaser.js. It handles the visuals, character interactions, and all the in-game action. When the player uses their Rubik’s cube, TensorFlow.js detects the cube, crops the face, and sends it to the backend to read the tile configuration.

The backend runs on Google Cloud Run, split into three services. The main service handles client requests and checks cube images using OpenCV for a first pass. If confidence is low, it calls a Gemma model for a second opinion. This two-stage system balances speed and accuracy.

Another (smaller) LLM service generates enemy “taunts” during combat, bringing the characters to life in ways scripted dialogue never could.

Challenges I ran into

Game engines. This was my first time building a game with a game engine, let alone Phaser.js, so that was an ill-advised challenge in itself!

Machine vision. Figuring out the Rubik’s cube configuration was another puzzle (heh). I first tried streaming the whole video feed to the backend, but the bandwidth cost and latency were brutal. Then I switched to streaming just the cube. TensorFlow.js isn’t trained to detect cubes, so we had to assume anything recognised as a cup/suitcase/phone/etc. was actually a cube. After a lot of testing and iteration, the solution was to prompt the player to hold up their cube for two seconds, snap a picture, and send that to the backend. It’s easier to crop, preserves privacy, reduces bandwidth, and makes the backend job faster.

Design. Well, let’s just say there may be a few crimes against good taste in there. It’s not my strongest suit. I usually get there eventually - but it takes a few more iterations than I've had.

Infrastructure. It took a bit of time to figure out how to set up the backend API, connect to Google Secret Manager, etc. Then, obviously I didn't want to expose my LLM's to the open internet, so I needed to ensure that the API could talk to the LLM - and that authentication piece took a bit of thought.

Accomplishments that I'm proud of

I'm pretty stoked with the speed of it all. From the Rubik's cube analysis works to the combat scene dialogue, it's quite a bit faster than I thought it might be.

What I learned

A lot! I learnt a lot about the trade offs of using a game engine vs standard web development. You get a lot with a game engine, but lose a lot of what I've taken advantage of with HTML and CSS wrt object positioning, responsivity, text layers (and Google Translate making websites accessible to others), etc.

It's been fascinating to see the ins and outs of how open source LLM's can be self hosted / cloud hosted. I've used the big cloud provider API's a bit, but outside of interpreting the responses or constructing prompts that give you what you need, I've not given much thought to what's involved in hosting a model.

What's next for SnailMail

The design was put together in a bit of a rush - there were a lot of moving parts. The first thing is to tighten up the look and feel. After that - it's a game that is easily playable on mobile, at least in principle. But it'll need some work to make the different scenes and controls work on smaller screens. I also have a bunch of fun ideas on making this multiplayer. Oh. Also, I'd like to train a dedicated model for TensorFlow.js to use to detect the Rubik's cube!

Built With

• flask
• google-cloud-run
• google-secret-manager
• opencv
• phaser.js
• python
• tensorflow.js
• typescript