Lost & Found

Inspiration

The inspiration stems from the desire to master a language and engage effortlessly in conversation. Initially, learning a new language can feel overwhelming, as if you are lost. Our aim is to transform language learning into an enjoyable and practical endeavor, with immersion serving as the pathway to achieving this objective. It emphasizes everyday language use, resulting in better and longer-lasting retention. Yale University confirms that immersion is the most effective method for language acquisition. Therefore, we have created an immersive mixed reality experience that transforms language learning into an enjoyable journey. Through this interactive platform, you'll learn a language by assisting others, honing your conversational skills in the process. While our current focus is on French, our vision extends to encompassing a diverse array of languages and cultural experiences. Why the name Lost and Found? Learning a new language may seem overwhelming at first, as if you are feeling lost. Our goal is to make the journey of language learning enjoyable and practical, and immersion can help you find that goal.

What it does

"Lost and Found" immerses you in a mixed reality language learning game where you assume the role of a customer service representative, assisting a consumer in the language you're studying. This interactive experience enriches language learning by offering real-time practice and enhancing your speaking, listening, and reading skills.

In this demo, our user is learning French and is tasked with guiding the customer through making ratatouille. The consumer's voice is AI-generated, providing a dynamic experience. Refer to the transcript on the monitor for additional support and interact with objects on your desk during the call. You'll have the ratatouille recipe to guide you through the conversation, and translations of unfamiliar words are available if needed. Once this call concludes, be prepared for another scenario with a new customer.

"Lost and Found" is designed to facilitate learning how to speak, listen, and read the language you wish to acquire. Through this immersive experience, you not only gain language proficiency but also insight into the culture. For instance, in this scenario, you learn how to make ratatouille. Lastly, "Lost and Found" imparts valuable lessons on assisting others, fostering a sense of empathy and community.

How we built it

"Lost and Found" is built using the Meta Presence Platform plugin within Unity, in conjunction with Meta Quest 3. We integrated various tools and SDKs, including the Interaction SDK, Open AI, Meta MR Utility Kit, Meta XR Interaction SDK, Scene API, Passthrough API, and Meta XR Voice, into Unity to create the immersive experience. Sound effects and music were polished using Audacity, while 3D assets sourced from online platforms underwent adjustments in Blender. Bezi and animations were employed for prototyping and visualizing interactions with each object. Figma played a crucial role in crafting UI info cards and branding elements, ensuring project cohesiveness. Additionally, we fostered open communication and a sense of enjoyment among team members throughout the development process.

Challenges we ran into

For the project, there were many ideas we wanted to explore and features to create in this experience. Something we slowly realized was how much we had to narrow our scope in. We had to really focus on what was key to this experience. Other challenge that we had, was finding a name for the experience, after a lot of research and talking, we decided on “Lost and Found.”

In terms of technical issues, a lot of time was spent getting familiar with building on Meta Quest 3 with the latest Meta XR SDK in Unity, including hand tracking/interactions, text-to-speech/speech-to-text. We also hosted a dedicated server for LLM communication and integrated it into our Unity project to provide a seamless experience.

Accomplishments that we're proud of

We take pride in our cohesive team dynamic, where diverse skill sets seamlessly merge to foster a unified approach to our work. With mutual respect for each other's needs and ideas, we collaborate effectively to achieve our goals.

Together, we've developed a Mixed Reality application that transcends traditional language learning methods. Our immersive platform enables users to engage in interactive learning experiences, such as conversing with an AI-generated voice, accessing conversation transcripts for enhanced comprehension, and seamlessly translating between French and English. Additionally, users can hear correct pronunciation of words in French, interact with virtual objects, and enjoy background music that immerses them in the culture, facilitating a comprehensive and enjoyable French language learning experience.

What we learned

• Customizing the Meta Presence Platform for our use case
• How to integrate Meta Presence Platform in Unity for custom hand interactions, hand tracking, scene, voice for spatial recognition and scene interaction in mixed reality
• Integrating Open AI
• Use of Bezi as a prototyping tool for interactions and collaboration
• Blender organic modeling, export GLB to Unity
• Splitting jobs up to ensure action items will be completed
• Working at an efficient pace to create an experience with a cross functional team

What's next for Lost and Found?

With Lost and Found, we've currently focused solely on a single customer scenario, but we aspire to broaden its scope by incorporating multiple customers with diverse experiences. As users progress, the vocabulary challenges will increase in difficulty, ensuring continuous learning. Additionally, we aim to introduce support for additional languages beyond French. While our demo is currently limited to French, allowing users to select the languages they wish to learn would enhance inclusivity. Our vision is to make language learning accessible and enjoyable for all, empowering individuals to engage in real-world conversations confidently.

Built With

• metaquest
• openai
• unity