PocketRealm

Your room, their playground. Welcome to Pocket Realm

Inspiration

Have you ever looked around your room and imagined you were somewhere else? The walls become a jungle, a tiny creature lives beside your desk, and an ordinary afternoon turns into an adventure.

PocketRealm grew out of that feeling. We wanted to bring the worlds we imagined as kids into the spaces we live in and let players shape those worlds with their own creativity.

What it does

PocketRealm turns your room into a personalized AR adventure. Scan your space with your iPhone (which is equipped with a LiDAR sensor), choose a theme, and discover a world built around your actual surroundings.

Your walls can take on jungle, coral reef, crystal cavern, or woodland decorations. Meet one of nine original characters, follow clues around your furniture, recover lost keepsakes, and earn friendship badges. Some friends also invite you to solve puzzles, from mixing a recipe to repeating a growing sequence of movements.

You can also draw something on paper and bring it into the adventure. The drawing workshop turns your sketch into a 3D sidekick, tool, or object you can use to overcome obstacles. Your creations stay in your inventory for future adventures.

Rooms can be saved and revisited without generating a new plan, and an ElevenLabs voice integration lets you talk with our fun little creatures!

How we built it

We built the native iOS app in Swift and SwiftUI, using Apple’s RoomPlan and LiDAR to capture room geometry, ARKit to track the space, and RealityKit to render characters and decorations.

Gemini interprets the measured layout and camera snapshots to plan decorations, character locations, clues, and greetings. Muse is also available as an alternative provider. The app validates each plan against the scanned room and handles the placement and game rules.

We created nine original characters and the themed environment assets in Blender, exporting them as USDZ models. Procedural animations give the characters greetings, playful reactions, and celebrations.

For drawings, a Python service passes the sketch and quest context to Codex, which builds a model in Blender and assigns its role in the game. The app downloads the finished asset and brings it into the room.

We also built an experimental Meta Ray-Ban Display integration for quest cards and a prototype character view.

Challenges we ran into

Making virtual objects feel like they belong in a real room required careful spatial work. Characters need to stand on surfaces, decorations need to fit the walls, and placements need to remain consistent as the player moves.

Connecting AI creativity to a playable game was another challenge. We constrained generated plans to measured locations and supported puzzles, then validated the results before starting an adventure.

We also had to handle interrupted tracking, generation delays, and saved-room recognition. Preserving progress and offering recovery paths became an important part of making the experience usable.

Accomplishments that we're proud of

We’re proud of connecting room scanning, personalized quests, original characters, and drawing-based creation in one app.

The drawing workshop is especially exciting: a sketch becomes something with a purpose in the adventure. Players can contribute their own ideas to the world they explore.

We also built reusable room saves and a shared animation system that gives all nine characters distinct personalities.

What we learned

We learned that convincing AR depends on small details such as a character’s feet touching the table, decorations respecting a doorway, and objects staying where you expect them.

We also learned how to combine generative AI with dependable game logic. AI can interpret a room and suggest a story, while the app keeps placement, puzzle rules, and progression consistent.

What's next for PocketRealm

We want to make PocketRealm a way to explore what you imagine in the space around you. Whether you’re bringing a sketch to life, reimagining your room, or trying out an idea, you should be able to see and interact with it without having to physically build it first.

That means expanding beyond our current adventures: giving creations more behaviors, making drawing generation faster, and letting users experiment with how their ideas look and fit into real environments. Better room recognition and a more natural glasses experience will help make that possible. Our goal is to shorten the distance between “what if?” and seeing it in front of you, giving people room to create, experiment, and explore possibilities before committing time, space, or materials.