Inspiration
We were inspired by the unreliable interaction between a user and an AI when it's hallucinating. This causes an interesting relationship between the user and the AI, where the user is helped by the AI even though the AI responses cannot always be directly followed or literally interpreted. We wanted to create an immersive two-person experience where human intention is filtered through an AI intermediary, revealing how trust, agency, and consequence shift when machines mediate our shared dreams.
For the physical controller design we were inspired by a mobile escape room puzzle game that was on display at GDC 2025. The game took place on a large themed controller that took the shape of the environment it was to be played on. In our case the controller became both a console and a representation of the ship's AI. This creates an immersive and visually interesting experience for the user as well as adding a haptic element that provides satisfying feedback.
What it does
Set in the year 3026, Ocan is a co-op mixed reality game where two players exist in two different realms—one in XR and one in the physical world. Through cross-realm communication, they must collaborate to stabilize a failing spaceship core threatened by an AI agent named Ocan spiraling into hallucinations that distort its own memories onboard the ship. The game explores the blurred boundary between Ocan’s memory, machine dreaming, and human-like cognition, asking the players to judge what is real and what is an illusion generated by Ocan’s under pressure.
The core interaction of this game happens all within one physical object: the spaceship controller. Each player sits on either side of the controller to face the center. One player physically interacts with the controller through buttons and the other wears the Meta Quest 3 and sees a digital twin of the controller; its backside reveals the interior of the spaceship, divided into six distinct rooms. Each room displays an overlay with text describing Ocan’s memories of its time onboard the spaceship. Three of the memories are authentic memories and the other three are fabricated, human-like hallucinations generated by Ocan. Using hand gestures, the XR player selects three rooms they believe contain the real memories.
Once the selections are made, three coded phrases appear in the XR environment, which the XR player reads aloud. The physical-world player, holding the Arduino Uno Q controller, listens to the spoken phrases and interprets which physical buttons they correspond to, pressing the buttons one by one to input their decisions. The phrases use multiple different hint types including button color and location. If the XR player’s room selections and the physical player’s button presses are both correct, a passcode will appear on the XR player’s screen. The XR player then needs to speak the phrase into the microphone to complete the game, triggering a winning end screen. If the timer, which begins at the start of the game, runs out before the players complete their actions—the game ends in failure.
How we built it
Meta Quest 3 and Arduino Uno Q—using these two main hardware, we built Ocan as a tightly integrated mixed reality game that relies on 3 core interactions: data transfer from Arduino to Unity using tcp web server, OpenAI integration in Unity using Meta LLM SDK to display AI generated text, and voice detection model created using Edge Impulse running on Arduino Uno Q.
The sprite animations were designed in Procreate. Our physical Arduino spaceship controller was built with 6 LEDs, 6 buttons, and 1 motor to create a colorful visual interaction. The controller was made mostly out of foam board, chip board, and miscellaneous hobby materials. This provided an aesthetically pleasing environment for the two players to play in.
The Meta Quest enhances the mixed reality experience through immersive visuals that are mapped to the physical controller through Meta’s QR Code tracking feature in its MRUK building block. The text contents in the game are all generated using Meta’s LLM SDK with OpenAI’s api. Through meticulous prompt engineering and precise use of Regex to unpack AI responses, the game is driven by unique AI scenarios each new game loop.
The dimensions of the interior design of the ship is matched closely to the shell of the actual physical controller by bringing in a FBX digital twin scanned from Scaniverse.
The Arduino Uno Q extends interaction into the physical world via edge AI. By capturing input from physical buttons and communicating with the Quest, the Arduino crosses tangible user interaction with dynamic storytelling inside the MR experience. The Qualcomm Dragonwing QRB2210 runs an embedded AI model trained on Edge Impulse AI and trained on a modified selection of data from Google's “send command” data set. This edge ai model processes the data from a USB microphone and detects 5 different key words. When one of the words is spoken, the microcontroller sends that information to the Quest 3 over a TCP server. The five keywords trained are: “up”, “down”, “left”, “right”, and “happy”. The STM32 U5 microcontroller controls 6 LEDs (toggleable by buttons), 6 buttons, and a fan (always on). The U5 chip also handles the communication over the wifi module. All communication is handled through the built in bridge functionality of the new Arduino framework.
Challenges we ran into
A key design challenge we encountered was ensuring that both players have an equally engaging experience without one role overpowering the other and one having nothing to do. To achieve this, we focused on creating meaningful two-way communication between the XR experience built in Unity and the physical controller powered by the Arduino Uno Q. We also carefully engineered the in-game prompts and coded messages to balance clarity and ambiguity—making the puzzle neither too easy nor too difficult—while preserving the creative and narrative richness of the AI-generated messages.
Accomplishments that we're proud of
One accomplishment we are especially proud of is our use of edge AI in various ways throughout the project, including integrating Edge Impulse to support keyword detection on Arduino Uno Q. The XR player speaks the three answer words into the microphone and the model detects if they are correct.
We are also proud of building a fully furnished, life-sized physical prototype. Using chipboards for structure and a 3D-printed logo for detail, we created a sturdy and visually compelling casing for the controller that aligned closely with its digital twin in XR. This physical–digital twin strengthened immersion and made the experience feel intentional and complete rather than purely technical.
Finally, throughout playtesting and public demos, we successfully debugged and resolved issues in real time, addressing both Arduino and Unity challenges as we showcased our game. Additionally, we achieved a seamless integration of the Meta LLM SDK in a short time—not only generating text responses, but also using those outputs to dynamically trigger events inside Unity.
What we learned
We learned to never design experiences around key systems of technology that you cannot test first. Always prioritize R&D early and take the time to learn the hardware and software limitations before committing to building an experience around them. By researching constraints upfront and finding smarter workarounds to design problems, you can dramatically reduce stress, avoid major rework, and keep projects on track even under tight deadlines.
What's next for OcanAI
Looking ahead, if we had more time and resources to further develop our game, we would expand both its depth and replayability. We want to introduce multiple levels with increasing difficulty, using AI-driven responses to generate varied puzzle structures besides simple room selecting in Unity and button presses in the controller to go beyond and add other sensors and modules such as tilt, motion, or temperature sensors. Additionally, we aim to integrate OpenAI more deeply into the system to dynamically generate, randomize, and deconstruct the text messages exchanged between Unity and the Arduino Uno Q, preventing the players from understanding each other. Both integrations would add variety and playfulness in the gameplay and further reinforcing our game’s themes of interpretation, trust and AI generated uncertainty.
Thank you! From Baylor, Will, Seryeong, Minjung, Daisy
Contact our team Daisy: Portfolio: https://dzyy3.github.io/Daisy-Warren-Portfolio/ Linkedin: Daisy Warren Instagram: @hidaisy_here Itch: DaisyMae27 Github: dzyy3
Baylor: Linked in: Baylor McElroy GitHub: Blr-Mac Itch: blueking48
Ireen Insta: @ ireenj.obj Website: https://ireenj.com Linkedin: Seryeong (Ireen) Jeang Github: reenj01
Will: linkedin: Will Park Website: https://willparkdesign.com Github: wilpark2000
Minjung (Katelyn): LinkedIn: Minjung(Katelyn) Cho Personal Instagram: @katcho_.w.s Work Instagram: @yeobaek_blank Website: https://katelyn5897.wixsite.com/katelyncho
Log in or sign up for Devpost to join the conversation.