Lucid Immersion

Inspiration

Hacking with new technologies is difficult, exhausting, yet also incredibly rewarding. As a team of four college students with little to no experience with AR/XR/VR devices, we entered this hackathon aiming to learn and, more importantly, build something beyond our individual capabilities.

When we first heard of the many tracks and sponsors at Immerse The Bay, one track stood out to us as being feasible and capable of long-lasting impact as the next generation of real-time assistive technology. That was Meta's Hands-On Coach track.

The Idea

Our idea for this hackathon was to augment a Meta Quest 3 with the ability to provide context-aware troubleshooting and general assistance, no matter the user's situation, by leveraging the new PassthroughCameraAccess component from the latest version of Meta's Mixed Reality Utility Toolkit (MRUK). Using the latest API version, we built a closed-loop feedback system that takes snapshots of the passthrough camera feed at user request, analyzes it, and crucially, reasons possible causes of user distress.

Our novel system finds crucial contextual details within images and stores them short-term. This makes subsequent responses for recurring issues both more efficient and effective while ensuring user privacy.

Our Takeaways

Starting early is NOT enough especially when using unfamiliar hardware. Our team was overwhelmed by the surplus of starter kits from each sponsor we wished to include in our project, and were made to understand firsthand the difficulties of compiling every api on a singular version of Unity.

Plan EVERYTHING Knowing the inputs and outputs of the project beforehand streamlines the process of development. But that is just the START of planning. Staging the project greatly increases the likelihood of having something presentable. Projects DON'T need to be perfect, hackathons are meant to be hacked together. Nearly all that matters is the vision and a short proof of concept.