Inspiration

I believe VR offers a unique opportunity for people to unwind, relax, and form meaningful bonds with virtual pets like Momo. Being able to interact with a pet through natural speech makes the experience feel much more personal and immersive.

Momo was also inspired by games I grew up playing, particularly Pokémon and Digimon World. What I loved about Digimon World was that raising and caring for your companion was just as important as battling. I wanted to bring that same sense of companionship into VR, while making battles more dynamic and engaging.

Rather than relying on traditional turn-based combat, I wanted players to give voice commands to their Momo in real time, almost like a tamer directing their companion during an actual battle. Instead of navigating menus to select moves, players can simply tell their Momo what to do and watch it respond. Voice is still relatively uncommon as a core gameplay mechanic, so I saw this hackathon as an opportunity to explore what it could bring to the experience.

What it does

Momo is a short VR demo where players can interact with their Momo by feeding them, playing with them, and taking them into battles.

During battles, Momo listens to the player's voice commands and acts accordingly. For example, shouting "Fireball!" commands Momo to launch a projectile fireball attack at its opponent. Rather than navigating menus or pressing buttons to select moves, players can simply call out commands as if they were actually directing their companion in battle. This makes combat feel more dynamic and creates a stronger sense of connection between the player and their Momo.

Beyond the pet experience, Momo is also an experiment in making voice and gaze first-class interaction methods on Meta Quest devices. A key focus was getting speech recognition and command processing to run locally on the headset with minimal latency.

By keeping voice processing on-device, Momo does not need to send player speech to external servers for recognition. This helps preserve user privacy while avoiding network round trips that could make real-time interactions feel sluggish. It also reduces dependence on cloud services and allows the core voice interaction experience to work without relying on an internet connection.

I developed and tested the demo on a Quest 2, with the expectation that newer Quest hardware could deliver even better performance.

How we built it

Momo was built using Unity and the Meta Quest Interaction SDK. The SDK made it relatively straightforward to implement hand tracking, gaze interactions, and gesture-based input.

For speech recognition, I experimented with several speech-to-text solutions, including Meta's Wit.ai. I eventually settled on Vosk because it offered a good balance between recognition accuracy and latency while running locally on the device. Since responsiveness was particularly important for real-time battles, I wanted to avoid the additional delay introduced by network requests.

An important design decision was to keep speech recognition on the headset rather than relying on cloud-based APIs. This means voice input can be processed without transmitting player speech to an external recognition service. It also removes network latency as a variable in the interaction pipeline.

Conversation voice lines were generated in advance using ElevenLabs, while other audio assets were generated locally using ComfyUI.

Challenges we ran into

Hand Interactions

Getting hand interactions to feel natural remains a challenge in VR. Grabbing virtual objects without physical or haptic feedback can sometimes break immersion. Throwing objects convincingly is also difficult, particularly when accurately detecting the moment a user releases an object and translating that movement into realistic physics.

Voice

Finding the right speech-to-text solution involved balancing recognition accuracy with latency. I tested several models before settling on Vosk.

I also experimented with running local LLMs on the Quest to allow players to interact with NPCs more dynamically, alongside text-to-speech for generating responses on the fly. The idea was to let players have more natural conversations with their companions, rather than being restricted to predefined dialogue. However, the latency was too high for the kind of fluid experience I wanted, so I decided to use prerecorded dialogue instead.

Another approach I explored was going directly from voice input to game actions using models that could interpret speech and make decisions. However, the accuracy was not reliable enough without further fine-tuning. Given the limited hackathon timeframe, I decided to focus on getting the core voice command system working reliably.

Game Mechanics

Building a complete pet-raising and battle system is a significant undertaking. Given the scope and time constraints of the hackathon, I focused on implementing the core gameplay loop and a basic real-time battle system with a limited set of moves and available Momos.

There is currently no full progression or levelling system, and the move selection is limited. These are areas I would explore if I continued developing the project.

Accomplishments that we're proud of

One of my biggest accomplishments was getting near-real-time voice-controlled battles working entirely on the Quest device, without sending voice data to external servers for recognition.

In my testing, speech recognition latency could reach as low as 50ms. This makes voice commands sufficiently responsive for real-time gameplay and demonstrates that on-device voice interaction can be practical even on standalone VR hardware.

Beyond the immediate gameplay experience, I see this project as a foundation for more advanced voice-driven, agentic interactions in VR. The same approach could be extended beyond predefined battle commands to allow virtual companions and NPCs to understand player intent, make decisions, and respond dynamically to natural speech. This opens up possibilities for more interactive and personalised VR experiences without having to rely entirely on cloud-based AI services.

I'm particularly proud of making privacy and responsiveness part of the core design rather than treating them as afterthoughts. Players can interact with their companions through voice without needing to rely on cloud-based speech recognition services.

I'm also proud to have brought an idea inspired by some of my childhood favourite games into VR, combining pet companionship with a more dynamic battle experience.

What we learned

I gained hands-on experience deploying and optimising machine learning models on Meta Quest devices, particularly the challenges of running speech recognition and exploring local LLM inference on resource-constrained hardware.

I also learned a lot about the practical trade-offs between latency, accuracy, and user experience when building voice-driven interactions for VR. Getting something to work locally is only part of the challenge. Making it responsive and reliable enough to feel natural during gameplay is another.

What's next for Momo

If there is sufficient community interest and support, I would love to continue developing Momo into a full game.

There is plenty of room to expand the pet-raising experience, introduce more Momos and abilities, and build a deeper real-time battle system.

On the technical side, I would also like to explore more advanced on-device AI interactions, allowing players to have dynamic conversations with their companions, teach them behaviours, and develop a more personal relationship with them over time.

Ultimately, I want to explore how voice and on-device AI can make virtual companions feel less like game characters and more like companions that players can genuinely connect with.

Built With

Share this project:

Updates

Submission history