Inspiration

We were initially inspired by Beat Saber and the immersive possibilities of virtual reality gaming. We wanted to explore whether we could bring some of that same interactive, immersive feeling into the physical world using a laptop and Meta Ray-Ban smart glasses, without requiring a traditional VR headset or controllers. As the project evolved, the gameplay moved in a different direction from our original concept, but the core idea remained the same: use wearable technology and natural movements to create a fun, interactive experience that bridges the gap between physical reality and virtual gaming.

What it does

MetaRayzer is an experimental hands-free gaming platform that combines Meta Ray-Ban smart glasses with a laptop to create an interactive gaming experience controlled through hand gestures and head movement. Our current minigame, Wizard’s Stand, puts players in the role of a wizard defending themselves against waves of fantasy enemies. Players cast spells using hand gestures, aim through movement, and react to incoming threats using defensive mechanics such as a shield. The game combines a live camera feed, hand tracking, visual spell effects, enemy encounters, and audio feedback to make the player's movements part of the gameplay. Rather than relying on a keyboard, mouse, or handheld controller for the core experience, MetaRayzer explores how smart glasses can become a new way to interact with games.

How we built it

We built MetaRayzer around a Flutter application that connects to Meta Ray-Ban smart glasses through Meta's Wearables Device Access Toolkit (DAT). The laptop application brings together the glasses' camera feed, hand detection and tracking, gesture recognition, and game rendering. We use tracked hand positions and gestures as gameplay inputs, translating physical movements into actions such as casting spells and activating defensive abilities. On the game side, we developed Wizard’s Stand with enemy behavior, combat mechanics, visual effects, and a progression system designed around these natural inputs. We also focused on making the camera feed and game interface work together so players can navigate menus and interact with the experience while using the glasses. The project has been developed iteratively, allowing us to refine the controls and gameplay as we learned more about the hardware and its capabilities.

Challenges we ran into

One of our biggest challenges was discovering that the camera's field of view and behavior when accessed through the API differed from the experience provided by the glasses' native hardware. This affected our original gameplay concept, which depended on a particular view of the player's surroundings. Rather than forcing the original idea to work around this limitation, we pivoted toward a game that could be played primarily with one hand, reducing dependence on the problematic field of view while preserving the immersive, movement-based interaction we wanted. We also had to work through the challenges of connecting wearable hardware to a desktop application, translating camera data into reliable hand tracking, and making gesture-based controls responsive enough for real-time gameplay.

Accomplishments that we're proud of

We're especially proud of integrating Meta Ray-Ban smart glasses with a laptop-based gaming experience and turning the glasses' camera input into an interactive control system. Bringing hand detection and tracking into the game was another major accomplishment, allowing physical gestures to drive gameplay rather than simply serve as visual input. We're also proud of adapting our original concept when we encountered hardware limitations. That process helped us create Wizard’s Stand, a more focused game concept that demonstrates the potential of wearable technology as an alternative gaming interface.

What we learned

We learned that building for emerging hardware requires more than just getting an API connection working. The actual behavior and limitations of a device can differ significantly from what a developer expects, so testing early and validating assumptions against real hardware is essential. We also learned how challenging it can be to turn raw camera input into reliable, low-latency interactions that feel natural to a player. Most importantly, we learned that technical constraints can lead to better design decisions: by adapting our gameplay to the hardware instead of fighting its limitations, we found a more practical way to explore hands-free gaming. The project also taught us the value of modular development, iterative testing, and keeping the user experience central to technical decisions.

What's next for MetaRayzer

Our next goal is to make MetaRayzer more polished, reliable, and expandable. For Wizard’s Stand, we want to improve hand-gesture recognition, refine camera-feed integration and audio output, enhance enemy designs and spell effects, and make the controls feel more responsive and intuitive. We're also exploring richer gameplay through interactive tutorials, new enemy types, boss encounters, defensive mechanics, and unlockable spells. Beyond the current minigame, our longer-term vision is to turn MetaRayzer into a platform for multiple games and experiences built around smart glasses, hand tracking, and natural movement. Ultimately, we want to explore how everyday wearable devices can offer a new kind of gaming experience without requiring dedicated VR headsets or traditional controllers.