RoomShift

A multiplayer AR escape room game with real time object-detection. Work with a team of 2-3 players using house hold objects as mission critical tools to solve puzzles in an AR escape room.

Comment

RoomShift: Blackout Protocol

Your home is the escape room. Everything in it could be the key.

Inspiration

We wanted to reimagine what an escape room could be by bringing the experience into players’ own homes. Traditional escape rooms rely on physical sets, dedicated props, and carefully designed environments. We asked: What if the room you were already standing in could become the game?

That question inspired RoomShift, a web-based augmented reality escape room concept where ordinary household objects become part of a science-fiction mission. We wanted to combine real-world exploration, computer vision, immersive 3D environments, and cooperative puzzle-solving to blur the boundary between the physical and digital worlds.

Our scenario, Blackout Protocol, has a facility undergo shutdown, causing room lockdowns. Players must investigate their surroundings, solve object-based puzzles, and restore the facility’s systems to escape. Although we did not complete the entire gameplay loop, building the prototype challenged us to tackle the difficult integration work behind this idea.

What It Does

RoomShift is a prototype for a mobile-first AR escape room designed to turn a player’s physical surroundings into part of the game.

In Blackout Protocol, everyday objects such as a book, cup, and set of keys serve as puzzle elements. The intended gameplay revolves around identifying these objects and arranging them in the correct sequence to progress through a virtual research facility.

We developed the frontend and explored how camera-based object recognition, 3D rendering, and a backend game state could work together. However, we were unable to complete the end-to-end experience during the hackathon.

Two major issues prevented us from reaching a fully playable game:

  • Camera-based object detection: We could not get object recognition working reliably enough to trigger progression to the next level. As a result, the main object-based puzzle could not consistently advance the game.
  • Backend deployment: We were unable to get our backend deployed successfully. We could see changes in our local development environment, but we could not get the deployed experience working as intended.

These limitations meant that the complete multiplayer escape-room flow was not finished by the end of the project.

How We Built It

We approached RoomShift as a combination of frontend development, computer vision, 3D graphics, and backend integration.

  • React, TypeScript, and Vite: Used for the frontend application and development workflow.
  • Three.js: Explored for rendering the virtual research facility and its 3D elements.
  • Web-based AR technologies: Explored as a way to bring a virtual environment into the player’s physical surroundings through a mobile browser.
  • Camera-based object recognition: We worked with Google's MediaPipe toward identifying everyday objects through the phone’s camera so that physical interactions could drive puzzle progression. Connecting recognition results reliably to the game’s level state remained unfinished.
  • DeepSpace backend: Intended to support shared game state and multiplayer coordination. Backend deployment issues prevented us from completing this integration in a working deployed experience.
  • Progressive Web App approach: We designed the experience around browser-based access, aiming to make it easier to try on a phone without installing a native app.

Our goal was to connect these pieces into a single gameplay loop: recognize a physical object, validate the puzzle action, update the game state, and unlock the next stage. We learned that making each component part of one reliable system was significantly more challenging than building the components separately.

The Intended Gameplay

1. Enter the facility

Players begin a mission inside a fictional research facility experiencing a blackout.

2. Find real-world objects

Players use their surroundings to locate household objects that serve as clues or puzzle pieces.

3. Solve the object puzzle

The first planned challenge uses the sequence BOOK → CUP → KEYS. The camera-based scanner was intended to recognize the required objects and allow the game to advance when the puzzle conditions were met. We were unable to get this detection-to-progression loop working reliably.

4. Restore the facility

Later stages were designed around virtual terminals, restoring power, and solving additional challenges. These stages depended on reliable puzzle progression and backend game-state integration, so we did not complete the full sequence.

5. Escape together

Our long-term gameplay concept includes shared progress and cooperative escape-room mechanics. We did not finish a fully deployed multiplayer experience during the hackathon.

Challenges We Faced

Getting object detection to drive gameplay

Our biggest gameplay blocker was connecting camera-based object detection to the game’s progression system. Recognizing or attempting to recognize an object is only one part of the problem. The application also needs to interpret the result correctly, verify that the puzzle conditions are met, and advance the player to the next level. We could not get this full interaction working reliably, which prevented us from completing the core puzzle loop.

Deploying the backend

Our second major blocker was backend deployment. While we could see and test changes locally, we were unable to get the backend deployed successfully for the intended experience. This meant we could not validate the complete application in its deployed environment or finish the intended backend-dependent functionality.

Integrating multiple systems under time pressure

RoomShift combined several challenging areas: a mobile web frontend, 3D rendering, camera access, object recognition, puzzle logic, and backend services. Getting these pieces to work together required more time than we had available. A component appearing to work on its own did not guarantee that it would work when connected to the rest of the application.

These challenges changed our understanding of what it takes to build an AR game. The hardest part was not just creating an immersive concept; it was making every technical layer communicate reliably enough to support actual gameplay.

What's Next

If we continue developing RoomShift, our priorities would be:

  • Make camera-based object recognition reliable enough to trigger puzzle progression.
  • Add clear debugging and fallback tools to help distinguish camera, detection, and game-state issues.
  • Deploy and verify the backend before building more features that depend on it.
  • Connect puzzle events to a consistent game-state system.
  • Test the complete experience on real iPhones and in the deployed environment.
  • Complete and test the progression from the first object puzzle through restoring power and escaping.
  • Improve the AR experience, mobile performance, and cooperative gameplay once the core loop is reliable.

What We Learned

RoomShift pushed us to work with technologies and integration challenges that were new to our team. We learned that an ambitious idea can involve much more than its visible interface: camera permissions, detection reliability, game-state transitions, backend deployment, and mobile-browser behavior all affect whether a game is actually playable.

We also learned the importance of validating the smallest complete gameplay loop early. Getting one object recognized, one puzzle condition verified, and one level transition working end to end would have helped us identify integration problems sooner. Likewise, deploying a minimal backend early would have exposed deployment issues before so much of the experience depended on it.

Although we did not finish the complete game, the process gave us a clearer understanding of the engineering required to turn an AR escape-room concept into a reliable experience.

The Vision

RoomShift is built around one simple idea: your surroundings should be part of the game.

We still believe that ordinary household objects can become mission-critical tools in an immersive escape room, without requiring a dedicated venue or expensive physical props. Our prototype represents an early step toward that vision—and a reminder that bringing the physical and digital worlds together requires both creative design and dependable engineering. Thank you to our wonderful team for trying to do something new! Jenil Patel, Ryan Ngo, Reagan Crockett, and Andel Budinich

Built With

Share this project:

Updates

Submission history