MagicPals

Screenshot of real product
Screenshot of real product
concept design of Lumi
development record
development record

Inspiration

Following the release of the Meta Quest 3 and Vision Pro, we have clearly seen that XR technology is rapidly becoming a mainstream method for spatial computing and human-computer interaction. Particularly within the realm of gamified applications, XR technology offers an unparalleled sense of immersion and interactive experience, allowing users to genuinely experience the wonderful sensations brought by the integration of virtual and real worlds. The creative inspiration for MagicPals stems from the possibilities of this emerging technology and our vision for the future of interactive experiences.

Seamless transition from daily spaces to fantastical worlds: We believe that future XR experiences should not be confined to traditional gaming modes but should be more integrated into everyday life, seamlessly transitioning from users' daily living spaces to expansive virtual worlds. MagicPals is designed with this philosophy in mind, allowing players to start in their own rooms and, together with their AI companion Lumi, explore more exciting virtual scenes.
Exploring emotional depth: As AI technology evolves from large general models to more specialized vertical application models, opportunities to provide deep emotional companionship to users are increasing. Lumi in MagicPals is not just a pet for players but a companion capable of understanding and responding to players' emotions. This depth of emotional interaction is what sets MagicPals apart from other virtual pet applications.
Pioneering new possibilities with XR technology: MagicPals leverages the latest technology from Meta Quest 3, such as spatial anchors, interactive SDKs, and passthrough technology, enabling players to interact with their virtual companion in their real environment, experiencing unprecedented interactive gameplay. The integration of these technologies not only advances the gaming experience but also opens new possibilities for the entire XR application field.

What it Does

MagicPals is an innovative AI-driven pet interaction game that creates a brand-new gaming experience using mixed reality technology. In MagicPals, players unexpectedly encounter a magical creature named Lumi, who is not just a virtual pet but an intelligent companion with its own emotions and memories.

Deep AI interaction: Lumi can understand and respond to players' voice commands through natural language processing technology. Its AI-driven personality and reactions evolve continuously based on player interactions. Players can chat with Lumi, share daily experiences, and even receive emotional feedback and suggestions from Lumi.
Integration of physical and virtual interactions: Utilizing the spatial localization and gesture recognition technology of Quest 3, players can physically interact with Lumi in their real environment, such as lifting, throwing, or gently patting, enhancing the intimacy between the player and Lumi.
Environmental exploration and gameplay mechanics: Players can interact with Lumi not only in their own rooms but also travel through portals to explore external fantastical worlds with Lumi. In these worlds, players and Lumi can participate in mini-games or collect rare items to decorate their real-world living spaces.
Ongoing game engagement: MagicPals is designed with a series of achievements and upgrade systems to encourage long-term player engagement. As players interact with Lumi, they can unlock new dialogue options, game functions, and virtual worlds, keeping each play session fresh and exciting.

How We Built It

In building MagicPals, our development team employed a series of advanced technologies and innovative methods to ensure our game not only delivers a unique player experience but also fully utilizes the advanced capabilities of Meta Quest 3 while ensuring high-quality implementation.

Utilizing advanced features of the Presence Platform: We chose Unity as our main development environment because it offers extensive support and a toolset that enables efficient development of mixed reality applications. By integrating features of the Presence Platform, such as Passthrough, spatial anchors, and the Interaction SDK, we created a true mixed reality experience that allows the virtual creature Lumi to move and interact freely in the player's real world.
Innovative interaction design:The core of MagicPals' interaction design lies in its innovative use of mixed reality, achievable only in this medium. We designed a series of location-based and gesture-driven interactions that allow players to interact naturally with Lumi, such as petting, throwing, and directing, along with transitioning effects that dissolve between the player's real space and virtual environments, and using virtual items to interact with the pet in the real environment. These interactions not only enhance the game's immersion but also fully demonstrate the unique applications of MR technology.
Performance optimization and platform compatibility:During the development process, we focused particularly on optimizing game performance to ensure optimal performance on Meta Quest 3. We meticulously adjusted our game's resource management and rendering settings to adapt to the characteristics of the Quest hardware, ensuring high frame rates and low latency even in the most complex scenes.

Challenges We Ran Into

Developing an engaging, novel, and easy-to-use gaming experience on the XR platform required nearly four months of design and repeated refinement. We also drew inspiration from some excellent works on Quest, such as The World Beyond and First Encounter.
We aimed to enrich the game experience through MR technology, which brought its own set of challenges. For example, designing weapon shooting mechanics without using controllers was a challenge; while traditional designs could simulate shooting with haptic feedback through controllers, this wasn't possible with hand-tracking alone.
In an MR environment, we hoped to use AI to enable the pet to interact in real-time with real objects in the scene, such as recognizing a water cup placed on the floor by the player and prompting a response from Lumi asking, "What is this? I've never seen it before." We believe this type of interaction can bridge the gap between players and virtual pets, breaking down the barriers between virtual and real.
There was intense debate about whether the player's companion should be more humanoid or non-humanoid, leading to multiple adjustments in the original design. Creating an IP character that leaves a lasting impression on players proved to be a challenging yet crucial task.
Designing certain special effects in the game was complicated by the inability to use full-screen post-processing effects when passthrough was enabled. This limitation resulted in reduced final image quality. Our compromise was to adjust each object's own glow, which, while effective, increased the workload. We hope to utilize post-processing effects in the future.

Accomplishments That We're Proud Of

We utilized several modules from the Presence Platform, including Passthrough, spatial anchors, and the Interactive SDK, to build our game experience and create many interesting gameplay elements.
We are in the process of creating a truly companionable pet game, unlike games that require players to actively care for pets. Lumi can establish a friendship-like relationship with players, engaging in quiet conversations in the room or jointly tackling challenges in the game. We hope to provide players with the motivation to continue playing over the long term.
During the AI-driven pet development process, we implemented memory saving. Each time a player logs in, Lumi's memories are restored. The continuity of memory, when Lumi recalls previous conversations, brings significant surprises to players.

What We Learned

Integration and Application of XR Technology: We learned how to apply XR technology to game development, especially how to integrate the features of Meta Quest 3's Presence Platform, including Passthrough, spatial anchors, and the Interactive SDK, within the Unity environment.
AI and Natural Language Processing: We gained deeper insights into enhancing interactions between users and AI pets using AI technology and natural language processing. Developing an AI pet that can remember player interactions and conversations was a valuable learning experience.
User Experience Design Challenges: In designing MagicPals, we deeply understood how to create a game experience that meets players' expectations and continues to attract them. Especially in designing a companion that blends humanoid and non-humanoid elements, we learned much about the importance of character design in forming emotional connections with players.
Technical Limitations and Creative Solutions: We faced some technical limitations, such as the inability to use full-screen post-processing effects in passthrough mode, which forced us to find other methods to ensure the quality of the game's visuals. This process taught us how to find innovative solutions within existing technical conditions.

What's Next for MagicPals

Deepening AI's Role in Companionship:
- Continuous Optimization of Emotional AI: We plan to enhance Lumi's ability to recognize and respond to players' emotional changes more accurately. Based on the player's current emotional state, Lumi will provide corresponding support or activity suggestions.
- Personalized Learning Systems: Lumi will use machine learning to understand players' preferences and behavioral patterns, allowing it to adjust its behavior and activities offered, better catering to individual player needs.
Deeper Integration of Augmented Reality and Virtual Reality:
- Integration of AR Elements: We will introduce augmented reality elements into the game, such as allowing Lumi to interact with actual household items. For example, Lumi could leave notes on the player's desk or "sit" on specific pieces of furniture.
- Real-Time Environment Mapping: Utilizing Meta Quest’s environmental understanding technology, Lumi will be able to react to real-time changes in the home environment, such as recognizing new furniture arrangements and freely moving around them.
Expansion of Game Content and Mechanics:
- New Exploration Worlds and Tasks: We will develop more fantastical worlds and themed tasks that offer new exploration challenges. These worlds will have unique environments and stories, encouraging players to adventure with Lumi, collect new items, and earn rewards.
- Multiplayer Mode: We aim to implement a multiplayer gaming environment where players can invite friends to join their world. Together with their Lumis, they can collaborate or compete in games, enhancing the game’s social interaction and replay value.