Inspiration
“Bring your ‘bored’ games to life.”
Living Table began with an idea we had been considering for at least two years: physical board games and digital play should feel connected. A familiar board, the pieces you already own, and the people around your table can become the foundation for something new.
What happens when your opponent has to leave, but you want to keep playing? Why should continuing that game mean abandoning its position or starting over?
Living Table: First Steps explores that question through chess. Our ambition is broad, but our starting point is personal: preserve the pleasure of playing at a real table while giving that experience new possibilities.
What it does
Living Table: First Steps is a hands-first mixed-reality chess alpha for Meta's new VR Glasses, Meta Quest 3, and the rest of Meta's extended MR family.
Players can align a virtual board with their physical chessboard, play against AI, continue the game virtually, and return to the physical table with the game’s progress preserved. Players can also start directly in virtual mode without a physical board.
The experience includes:
- Three AI difficulty settings with distinct playing tendencies.
- Hand interactions for moving pieces and positioning the board.
- Saved games and named captures of game positions.
- Visual guidance for moves, captures, check, and returning to physical play.
- Adjustable piece appearance, height, size, and opacity.
- An introductory setup tutorial and contextual help.
Physical play combines camera-based recognition with explicit player confirmation when needed. Recognition remains an area of active development; the alpha provides a manual path so uncertainty does not have to stop the game.
How we built it
All coding was carried out through GPT Astra, with human-led product design, art direction, and repeated testing on Quest 3. Design work for this competition began after the September 26 announcement, building on the earlier concept.
We developed the application in Unity using Meta’s XR SDK. Development was grounded in actual headset use: build locally, install on Quest, play, capture logs and camera footage, inspect failures, and refine the implementation.
The project evolved from a broader board-game platform into a focused chess experience. Chess gave us a precise environment for developing legal game-state handling, AI opponents, physical-to-virtual transitions, saved sessions, and spatial alignment.
We kept authoritative chess rules separate from camera observations. A camera reading supplies evidence about the physical board; it does not redefine what constitutes a legal move.
Challenges we ran into
Camera recognition was our largest technical challenge.
From a seated position, pieces overlap visually. Hands obscure squares, lighting changes appearances, and head movement changes perspective. Recognizing an empty square is not straightforward when a nearby king blocks the view. Missing visual evidence must not be mistaken for a missing piece.
We investigated camera input, projected board geometry, piece appearance, reference data, and the logic that turns observations into moves. We also introduced assisted confirmation to keep play usable when recognition was uncertain.
Spatial consistency was another major challenge. The physical board needed to retain its placement independently of the player’s facing direction and the movable virtual board, including after transitions and relaunches.
Other substantial work included sleep/wake stability, camera-analysis performance, saving without noticeable interruptions, readable spatial interfaces, and making hand interactions understandable to a first-time player.
Our hardware testing was concentrated on Quest 3. Broader hardware and environment validation remains ahead.
Accomplishments that we're proud of
We built a playable early alpha around a demanding interaction: continuing one chess game across physical and virtual play.
Through repeated headset testing, we reached successful physical-to-virtual-to-physical transitions, preserved placement across tested relaunches, and completed repeated sleep/wake checks without losing the game. We also improved responsiveness, reduced noticeable interruptions, and refined the interface through direct playtesting.
We are particularly proud of making the experience recoverable. Players can confirm uncertain physical moves, undo mistakes, save their progress, and keep playing.
Living Table’s intent is simple to explain, while its possibilities extend beyond chess. This alpha gives us a working foundation to explore those possibilities without pretending that every part of the larger vision is already complete.
What we learned
We learned to distinguish what a camera sees from what a system can confidently conclude.
Reliable physical interaction requires more than recognition alone. Camera timing, perspective, alignment, occlusion, reference quality, and game-state validation all contribute to the result. Clear feedback and graceful recovery are equally important.
We also learned that mixed-reality usability must be tested in the headset. Instructions that look readable on a monitor can be difficult to use while looking down at a board. Controls can obstruct real objects. Small placement errors become obvious when a player changes posture.
Finally, this project helped us develop an AI-assisted production workflow grounded in traditional game-development practices: human creative direction, iterative implementation, actual-device testing, instrumentation, and regression checks. AI accelerated the work; testing determined whether the experience worked.
What's next for Living Table: First Steps
Our immediate priorities are recognition reliability across more physical sets and rooms, independent first-time-user testing, and continued performance and accessibility improvements.
Beyond that, we plan to develop:
- Agents: more expressive opponents, optional coaching, and conversational assistance that respects the player’s control.
- Experiences: additional board games and interactions built on the physical-to-virtual foundation.
- Backend: dependable supporting services with a supported self-hosted path.
- Depth: richer personalization, refined materials and presentation, and more ways to revisit and explore saved games.
- - 3rd Party Licensors: (shhhhhh...) partner with publishers and brands to bring xxxx, yyyy, and zzzz into mixed reality.
The long-term vision is a platform that gives familiar tabletop experiences new life. First Steps is our playable beginning.

