Reality Quest

Turn your room into an AI-generated physical adventure.

Comment

Inspiration

Most webcam-based movement games assume that the player’s entire body is visible. In practice, laptop cameras often capture only the upper body, especially in small rooms. This makes many pose-based experiences difficult or impossible to use without moving the computer or stepping far away.

Reality Quest started with a simple question: what if the game adapted to what the camera can actually see, instead of forcing the player to adapt to the camera?

What it does

Reality Quest is a browser-based physical adventure game that turns a webcam and a small room into an interactive space mission.

The application detects which parts of the player’s body are reliably visible and automatically selects one of two capability modes:

  • Upper-body mode, which works with only the head, shoulders, elbows, and hands visible
  • Full-body mode, which unlocks additional movements such as squats

The player completes five physical missions by raising a hand, raising both hands, leaning, holding still, extending both arms, or placing their hands near their head. Each mission includes a timer, hold-progress indicator, score, streak, and story feedback.

The system only selects actions that the current camera framing can reliably evaluate. Upper-body players therefore receive a complete playable experience without ever being asked to perform a movement that requires visible knees or ankles.

Pose detection runs entirely in the browser. Camera frames are never uploaded to the server.

How we built it

Reality Quest is built as a full-stack Next.js application using TypeScript, React, Tailwind CSS, MediaPipe Pose Landmarker, Zod, and the OpenAI Responses API integration.

MediaPipe processes pose landmarks locally in the browser. A capability-detection module evaluates landmark visibility over multiple frames and uses hysteresis to prevent the mode from rapidly switching when landmarks are temporarily lost.

Each movement is implemented as a normalized and testable pose predicate. The predicates use body proportions such as shoulder width rather than fixed pixel distances, making them more robust across different camera resolutions and player distances.

The mission system uses a strict Zod schema and a constrained list of supportedActions. When a valid OpenAI API key is configured, the GPT-5.6 Responses API integration is designed to act as an adaptive game director, generating structured missions based on the player’s available movements, score, streak, difficulty, language, and recent story history. Model output is validated again before it reaches the game.

The public demo currently runs in clearly labeled Offline Demo Mode because no production OpenAI API key is configured. It uses a safe local mission generator and remains fully playable for all five missions. The GPT-5.6 runtime integration can be enabled through server-side environment configuration.

Codex with GPT-5.6 was the primary development environment for the project. It helped scaffold the application, implement the pose predicates, design the capability-driven mission architecture, add bilingual support, create the API validation layer, write tests, diagnose camera-loop issues, harden deployment, and prepare the judging documentation.

Challenges we ran into

The largest challenge was real-world camera framing. The first version assumed that knees and ankles would be visible, but testing on a normal laptop showed that only the upper body could be captured reliably.

Instead of merely lowering confidence thresholds, we redesigned the game around explicit pose capabilities. Mission generation, calibration, action detection, and the AI schema now share the same supported-action contract.

Pose-landmark jitter was another challenge. We added temporal smoothing, short landmark-loss tolerance, hold-duration verification, and multi-frame mode detection so that a single unstable frame does not reset the player’s progress.

We also hardened the public deployment by adding local-first MediaPipe assets with an official CDN fallback, resource cleanup, request cancellation, strict API validation, timeouts, offline fallback behavior, and accurate AI/offline status indicators.

Accomplishments that we are proud of

  • A complete five-mission game that works with an ordinary upper-body laptop camera
  • Automatic upper-body and full-body capability detection
  • Nine supported actions with capability-based filtering
  • Browser-only pose processing with no camera-frame uploads
  • Bilingual Chinese and English interfaces
  • Safe Offline Demo behavior when no API key is configured
  • Strict structured mission validation for the GPT-5.6 integration
  • 23 automated tests, production build validation, and GitHub Actions CI
  • A publicly deployed HTTPS demo that judges can open without rebuilding the project

What we learned

The most important lesson was that adaptive AI systems need an explicit understanding of what the surrounding system can actually execute.

Allowing a model to invent unrestricted movements would create missions that appear creative but cannot be evaluated. By constraining generation with real-time camera capabilities, the AI layer becomes safer, more reliable, and more useful.

We also learned that graceful fallback behavior is essential for interactive AI products. Reality Quest remains understandable and playable even when a model API, network connection, camera permission, or external asset is unavailable.

What’s next

The next step is to enable the production GPT-5.6 game director and expand its ability to maintain longer adventure narratives while remaining constrained by the player’s supported actions.

Future versions could add accessibility profiles, adjustable movement intensity, more visual themes, personalized difficulty, audio guidance, and additional safe movement categories.

Built With

Share this project:

Updates