TableConnect

Playing a game!
Real-time object detection overlays green 3D boxes on the physical board, confirming accurate piece and grid tracking.

Inspiration: A Game Night Lost to Distance

My passion has always been gathering friends and family around a physical board game. There's a tangible joy in handling the pieces and sharing a feeling that digital versions often miss. During the recent worldwide lockdowns, I tried to keep this tradition alive using online and VR board games, but the experience felt cold and disconnected. The tactile joy was gone. It was this frustration that crystallised the long-standing idea: why can't we have the best of both worlds? The time felt right to bring TableConnect a true mixed reality bridge for board gamers to life.

Building TableConnect

The execution of TableConnect required fusing computer vision, 3D development, and network programming. The core process started with the AI vision system, where we built upon an existing open-source checkers project on Roboflow for annotated images. We captured and annotated hundreds of pictures of our own checkers board, training and optimising our detection model from YOLOv8n to the efficient YOLOv12n. In parallel, my teammate created all the necessary 3D assets for the virtual game pieces and environment and was responsible for setting up the Unity scenes. We integrated the AI model into Unity using Sentis and ensured high performance via Threading and coroutines. The final AI output, converting the model's 2D bounding box into a 3D position in the user's room via raycasting and depth perception, was seamlessly integrated into the core game loop. My teammate then programmed the game logic, enabling piece movement and game states. Crucially, they set up the multiplayer functionality using Normcore, ensuring reliable communication and that the experience shown in the demo video functioned flawlessly.

Challenges: The Race Against Time

The development process was an intense sprint. Despite only discovering the project opportunity on November 7th, the intensely tight deadline was a welcome challenge. We eagerly embraced the chance to rapidly accelerate our learning, which included mastering advanced concepts like neural network optimisation and complex networking logic. Juggling this massive effort alongside full-time jobs required immense discipline. Specific technical hurdles included dealing with slow build times inherent to Unity MR development. A major challenge involved syncing game states between two players via the network such that each player could only move their own pieces, and preventing and resolving conflicts between contradicting game states across the network. Ultimately, we had to learn how to make MR apps and enable multiplayer under a highly compressed timescale.

Learnings: From Pixels to Position

This project was an incredible learning experience. We successfully learned how to set up a Unity scene to enable the Meta Quest’s MR features, like passthrough and how to set up a multiplayer game using Normcore. We gained practical expertise in training and optimising an image recognition model for mobile deployment. Crucially, we learned how to run the model on a texture derived from the Meta passthrough camera, and the complex process of using raycasting and depth perception to map a 2D bounding box detection into a precise 3D position in the real environment. The entire process, from data annotation to real-time spatial awareness, proved the viability of blending physical and virtual play, making TableConnect a true mixed reality achievement.

Future Improvements

To realise the full potential of TableConnect, my future plans include: Accuracy: Improve the Checkers image detection model for flawless performance. Game Expansion: Adding Chess and Backgammon options to the app, and supporting large-scale strategy and multiplayer cooperative games (which require 2 or more people). Social & Coaching: Implementing a feature to show the other person's avatar on the other side of the board for better social presence, and adding game tutorials and a coaching feature. Platform Polish: Developing more extensive opponent-finding options, including the ability to easily invite friends. Statistics: Adding detailed game statistics tracking.

Built With

c#
metaquest
passthrough
sentis
unity

Submitted to

Meta Horizon Start Developer Competition

Created by

I was solely responsible for the image recognition. This included data gathering and annotation, training and optimising the AI model, integrating it into Unity via Sentis, and implementing high-performance logic using threading. My key contribution was mastering the complex 2D-to-3D spatial mapping pipeline with Meta's raycasting to identify physical piece positions.

Anthony Cannon
I was responsible for setting up the core game logic within Unity; creating scenes and 3D assets, enabling multiplayer and programming the main game loop.
I also appeared in the video demonstrating the app.

Nazar Elsayed