Reality.exe: Escape Protocol

Inspiration

What if an ordinary FaceTime call could turn your surroundings into an escape room?

Traditional escape rooms require physical locations, while most digital escape games rely on clicking buttons and interacting with a screen. We wanted to combine the best of both worlds by creating an experience where the real world becomes the game itself.

Inspired by immersive escape rooms, sci-fi adventures, and the possibilities of real-time multimodal AI, we built Reality.exe: Escape Protocol—an AI-powered FaceTime escape room where players must save a malfunctioning spaceship using everyday objects around them.

Our goal was to make AI feel less like a chatbot and more like a character that can see, speak, react, and interact with the physical world.

What It Does

Reality.exe transforms a FaceTime video call into an interactive sci-fi adventure.

Players connect with ORBIT, a sarcastic, slightly panicked spaceship AI that needs their help to prevent a catastrophic system failure.

Instead of using traditional game controllers, players interact with real objects around them.

The experience follows a series of missions:

  1. Object Discovery: Players show three everyday objects to their camera. ORBIT identifies them and assigns spaceship-related roles, such as a reactor, control panel, or access key.
  2. Reactor Activation: Players must physically position objects in a specific arrangement to restore power.
  3. Security Override: Players complete a camera-based challenge to unlock the spaceship's security system.
  4. Escape Sequence: Players arrange their objects in the correct order to activate the escape portal.

Using computer vision, the AI evaluates whether players have actually completed each challenge. Players cannot simply say they finished a mission—they must demonstrate it through their camera.

Throughout the experience, an animated spaceship HUD displays mission progress and system status, making the FaceTime call feel like a live communication channel with a spaceship.

How We Built It

We designed Reality.exe around three interconnected systems: real-time communication, AI-powered vision, and interactive game logic.

Real-Time FaceTime Integration

We used Chert to connect real FaceTime calls to our custom AI agent and LiveKit to manage real-time audio and video communication.

Our integration handles:

  • Incoming FaceTime audio and video
  • AI-powered speech recognition and voice responses
  • Camera-frame capture for visual verification
  • Outgoing animated spaceship HUD video
  • Connection lifecycle and call cleanup

For the voice pipeline, we integrated speech recognition, language generation, and text-to-speech services through LiveKit Inference.

AI Vision with Grok

We chose xAI's Grok API for our computer vision system.

The vision component is designed to analyze camera frames and evaluate physical relationships between objects, such as whether one object is placed on another, an object is being held up, or multiple objects are arranged in a specific order.

To improve reliability, we designed a multi-frame verification approach that evaluates several observations rather than trusting a single image.

Game Engine and Visual Interface

We built the game around a deterministic mission system that manages objectives, verification results, and mission progression.

The game engine separates AI observations from game decisions, preventing the language model from arbitrarily advancing a mission.

We also developed a spaceship-themed HUD that visually communicates reactor status, security progress, and escape readiness.

Technologies Used

  • Chert — FaceTime connectivity
  • LiveKit — Real-time audio and video transport
  • Grok API (xAI) — AI-powered visual understanding
  • TypeScript / Node.js — Agent integration and game logic
  • Deepgram — Speech-to-text
  • GPT-4.1 mini — Conversational AI
  • Cartesia — Text-to-speech

Challenges We Faced

1. Connecting a Custom AI Agent to FaceTime

One of our biggest challenges was building a real-time AI experience on top of FaceTime rather than a traditional web application.

We needed to connect Chert's FaceTime infrastructure with our LiveKit agent, configure authentication, and manage incoming and outgoing media streams.

We initially encountered authentication failures and connection issues, which required debugging API credentials, agent configuration, and the Chert connector.

After resolving these issues, we successfully established a real FaceTime connection with our custom agent.

2. Working with Real-Time Video

Unlike a typical chatbot, our AI needs to understand what players physically do.

We had to handle live camera frames, distinguish separate observations, and design a system that could pass visual information to our vision model without interrupting the conversation.

We also needed to publish our own animated HUD back into the FaceTime call.

To validate the transport independently, we created synthetic LiveKit tests that successfully transmitted both HUD states and captured five distinct JPEG frames.

3. Making AI Verification Reliable

Computer vision can misinterpret object positions, particularly when lighting, camera angles, or object visibility are poor.

To address this, we designed a verification approach using multiple frames, confidence checks, and structured results.

We also separated visual interpretation from mission progression so the game engine—not the AI's conversational output—determines whether a challenge is complete.

4. Building Under Hackathon Constraints

We had limited development time and wanted to minimize API costs.

This pushed us to prioritize free-tier resources where available, develop independent components in parallel, and create mock tests so teammates could continue working without waiting for the full FaceTime integration.

What We Learned

This project taught us that building a real-time AI application involves much more than connecting an API to a user interface.

We learned how to:

  • Integrate real-time communication infrastructure with AI agents.
  • Handle bidirectional audio and video streams.
  • Design structured interfaces between independent AI and game components.
  • Build more reliable computer vision verification using multiple observations.
  • Separate probabilistic AI outputs from deterministic application logic.
  • Debug authentication, runtime compatibility, and media connections across multiple services.
  • Collaborate on a complex system under a tight hackathon deadline.

Most importantly, we learned how to make AI interactions feel more physical, immersive, and engaging.

What's Next

We envision Reality.exe becoming a platform for interactive, AI-powered experiences that blend the digital and physical worlds.

Future improvements could include:

  • Procedurally generated missions that adapt to whatever objects players have available.
  • Multiplayer escape rooms where friends collaborate through video calls.
  • Adaptive difficulty based on player performance and surroundings.
  • More immersive storytelling, with branching narratives and different AI characters.
  • Expanded environments, from haunted houses to detective mysteries and space adventures.

Our long-term vision is to make interactive AI experiences accessible through something people already use every day: a video call.

Reality.exe isn't just an escape room on FaceTime. It's an experiment in turning the physical world into an AI-powered game.

Built With

  • ai-agents
  • cartesia
  • chert
  • computer-vision
  • deepgram
  • facetime
  • gpt-4.1-mini
  • grok
  • javascript
  • livekit
  • livekit-agents
  • multimodal
  • node.js
  • real-time-ai
  • speech-to-text
  • text-to-speech
  • typescript
  • webrtc
  • xai-api
Share this project:

Updates

Submission history