[S6] - LocavociXR

setup object
show label raycast 1
show label raycast 2
show label raycast 3

Inspiration

The Memory Palace or Loci method is a useful tool for memorization. It works by associating words or concepts to a place, e.g. entering the living room prompts you to think about words for leisure. While this has proven effective, we believe XR can improve on the technique using digital objects, labels and a pinch of gamification. Also spaced repetition is made possible, by not prompting every object every time.

What it does

The prototype has three functionalities: Pointing at real objects and displaying the associated label, spawning interactable objects and language assistance. The prototype is aimed at english speakers wanting to learn swedish.

How we built it

During the hackathon we tinkered with different technologies, including the AI image capturing, the voice SDK, and even Sloyd to generate objects on the fly, although that didn't pan out. So we developed some pre-fabs to be able to simulate the repetition part of the Loci method, having the user spawn randomly selected objects from our list. Using raycasting we identify scanned objects in the room. For voice recognition we decided on using ChatGPT.

Challenges we ran into

For parts of the SDK (e.g. Voice) we struggled to make sense of the available documentation. As our team was comparatively inexperienced with XR development, the learning curve was pretty steep for us.

Accomplishments that we're proud of

We managed to implement the AI-toolkit and use it to understand and talk about objects found in our screenshots. Also, we got a base idea of our vision of a language learning application working. After the setbacks on the second day, we had a great moment of rejoicing when we got the object recognition up and running.

What we learned

One major timesink was getting the collaboration tools working, generally time management and making sure to align our vision. Albeit struggling with the documentation, we did get a better understanding of how to apply and use the building blocks.