Inspiration
We wanted to capture the feeling of tipping a box of building pieces onto a table and seeing what you could make. Mixed reality let us keep the familiar motions of picking up, turning, and joining pieces while making room for things physical blocks can’t do, like floating within reach while you line up the next connection.
That gave us a question for the AI, too. Could a companion feel like someone sitting beside you at the workbench, ready to help without taking the pieces out of your hands?
What it does
Your first job in Kliktopia is to rebuild a lighthouse after one of Robert’s experiments goes wrong. Robey talks you through your first few moves as you pinch pieces, line up their faces, and release to snap them together. Once the lighthouse is rebuilt, you carry it back to its spot on the rocky point.
From there, you can tackle blueprint challenges, earn pieces for your collection, display finished builds, or use your spare pieces in Free Build. The five difficulty modes range from Guided to Hardcore Grandmaster, where you study the finished model, then rebuild it from memory against the clock.
How we built it
We built Kliktopia in Unity and C# using Meta Quest passthrough and hand tracking. We designed smooth, beveled blocks with grid-based connection points on their faces. Underneath, a deterministic construction engine checks for overlaps, tracks how pieces connect, and verifies finished builds against their blueprints. Inventory and progress are saved on the headset.
For live conversation, we connected Unity to a Python backend using FastAPI, Pipecat, WebRTC, and OpenAI Realtime. Robey can read relevant game state and provide visual cues when appropriate. The tools available to Robey give it no way to move a piece, complete a build, or award a reward. Wouldn't be much of a game if it played itself!
Challenges we ran into
The hardest challenge was getting the hand interaction and manipulation feeling just right. We went through several iterations and spent probably 30% of the entire project making sure this felt great... Many many days testing and playing, meticulously tweaking parameters and adding and removing variables to manipulate results under certain conditions until eventually everything felt great.. If it wasn't fun and easy and intuitive to move blocks around and snap and unsnap them, then nothing else would matter.
Accomplishments that we're proud of
We made a first lesson that feels like a small rescue mission. The harbour goes dark, you rebuild its lighthouse piece by piece, and you carry the whole thing back into place. The same building controls carry through to challenges and Free Build, so the skills you learn with Robey are the ones you keep using on your own.
We’re also proud that a voice conversation can make the workshop feel more alive while the full building experience remains available offline.
What we learned
A snap is a promise. Once the game shows where a piece will land, the release has to feel consistent with that preview. Testing on Quest made it clear how much the experience depends on that last small movement.
We also learned how much a companion can do with a small set of reliable facts. Knowing what the player is holding and what just connected gives Robey enough context to answer a useful question. Sometimes the best help is one sentence and a moment of quiet to try it.
What's next for Kliktopia
We want to put the lighthouse rescue in front of more first-time Quest players and watch where they hesitate, reach too far, or lose track of the next piece. Then we want to grow the campaign with more models that do something fun after you build them.
We’re also prototyping an AI blueprint designer that can propose variations on designs and even create blueprints for new designs that the player asks for in-game.
Built With
- chatgpt
- codex
- unity