smrt tags

smrt tags: Bridging the Language Gap with Mixed Reality

Inspiration

Moving to a new country that uses a language not native to me can be daunting. My partner and I recently faced this challenge, and we found ourselves using sticky notes to label objects in our apartment with their names in the new language. These constant reminders helped us learn faster but also cluttered our home.

This experience sparked the idea for smrt tags, a mixed reality (MR) app that utilizes spatial translation tags. Imagine being able to place virtual labels on objects in your environment without visitors being able to see the tags placed everywhere– a coffee cup tagged as "café", a chair tagged as "silla". These persistent tags, accessible through your MR headset, would constantly reinforce your learning, similar to our paper labels.

Building smrt tags

The journey to smrt tags involved several key steps:

Planning and Scoping: We started with pen-and-paper planning, defining the project's overall vision while establishing a clear Minimum Viable Product (MVP). This plan was then transferred to Trello.com for efficient task management.
Tech Stack: Unity served as the development engine, while the Meta Presence Platform provided the foundation for the mixed reality experience. Passthrough technology allowed for a seamless blend of the real and virtual worlds, and spatial anchors facilitated persistent tag placement.
User Interaction: User interaction was designed to be controller-free, utilizing the Meta Interaction SDK. This allowed users to manipulate tags and access translations without needing handheld devices.
Testing and Delivery: Extensive testing with multiple users and in various rooms and locations ensured a smooth user experience. One of our team members was able to transform recorded gameplay footage into a compelling video presentation.

What I Learned

This project pushed me in several ways:

Prioritization is Key: Focusing on the MVP first allowed me to deliver a core experience on time. The additional features can always come later!
Read, Experiment, Repeat: Documentation isn't always perfect. While reading the resources provided was crucial, I found that experimenting with the code itself often yielded better results and uncovered unexpected solutions.

For example, the virtual keyboard wouldn't spawn in the editor as expected, but worked flawlessly during the built project. This kind of hands-on learning proved invaluable.

Challenges: Embracing the Unfamiliar

While I'm comfortable with openXR and developing for various headsets, this project utilized the Meta Presence Platform, which was new to me. Unlike openXR's broader headset compatibility, the Presence Platform is specifically designed for Meta Quest devices.

This platform's documentation, though helpful, wasn't as comprehensive as I'm accustomed to. This meant I had to delve deeper, directly exploring the Meta codebase and experimenting to overcome obstacles. The upside? By tackling these challenges head-on, I not only solved my own problems but also gained valuable insights that could assist others venturing into the world of the Meta Presence Platform.

Accomplishments

This project represents more than just lines of code; it's a testament to several achievements:

Personal Utility: We're most excited about the practical application of smrt tags. It's a tool we can use ourselves to actively learn the language in our home.
Embracing New Tech: Stepping outside my comfort zone and successfully integrating the Meta Presence Platform was a rewarding challenge.
From Plan to Prototype: The project journey, from initial planning to coding and testing, allowed me to put learned techniques into practice throughout the entire development cycle.
Future-Proofed Foundation: The codebase is designed with expandability in mind, allowing for easy integration of new features as we continue to develop smrt tags.
Exceeding Expectations: We not only completed the MVP, but also managed to add some additional features. While there's always room for refinement, we're incredibly proud of the final outcome

The Road Ahead

smrt tags represent a first step towards a more immersive and interactive language learning experience. We see this initial version as a springboard for exciting future features. One near-term goal is to enhance user engagement by incorporating a matching-type game directly within the tags. This will gamify the learning process, making it even more engaging and effective.

Looking even further ahead, we plan to explore features like voice recognition and pronunciation feedback. These functionalities would provide learners with real-time feedback on their spoken language, further solidifying their understanding. By continuously adding these layers of interactivity, smrt tags has the potential to revolutionize the way people approach language acquisition.