[U18] - Language Room

Labelling a virtual object
A screenshot is labelled
Selecting a mode
Labelling a virtual object

Inspiration

One of our developers is trying to learn German, and needed a little extra help. As avid XR enthusiasts, bringing linguistic learning to XR with AI just made sense!

What it does

In Language Room, you can pick from 8 languages to learn and test yourself with. We use a scan of your room (with Meta Mixed Reality Utility Kit) to spawn a few colourful objects for you to interact with.

In Learn Mode you can use Poke Interaction to select a virtual object which GPT4 will then translate into your chosen language as well as labelling it and reading it out. You can also use controllers to take a passthrough photo of anything in your real space, after which GPT4 will again translate and read the name of the subject in your chosen language.

In Test Mode a set of tasks is given to you. These are generated programmatically from the content availble in your environment. For example, you might be told "Go to the table" but this instruction will be read and shown to you in the language you have selected. Once you perform this task, another task will be given. You can use this mode to test your reading and listening comprehension.

How we built it

We used a combination of Meta building blocks to build out the space recognition using the Meta Mixed Reality Utility Kit, Passthrough and Anchor system to bring the application into your world. We used the Image Capture AI Kit alongside ChatGPT's translation capabilities to contextualise the objects in your real space into your desired language. Once we have the translation, we simply use the Text to Speech component included in the Image Capture package to read out the translation as well as display the text for a multimodal output experience.

Challenges we ran into

We initially used the Realtime Object Detection package and whilst we did get it up and running we found it wasn't really suitable for our project because of the dual connection dependency. We spent the first half of day 1 investigating both image kits but this did take time away from development. Our application is OpenAI and ChatGPT dependent for all of the translation. Because of the downtime, we were unable to test for a significant period during day 2, and built a backup system of a known library of German translations, which would be very limited compared to the AI system. Ultimately, we did not need to use this fallback.

Throughout the project we struggled with the integration of the Interaction SDK buttons. On day 3 we had to rebuild our entire UX to work around this.

Hardware failure. Dan's laptop had issues on day two, limiting what could be done until the hardware issues were resolved.

Accomplishments that we're proud of

Use any object in your space to learn and test yourself on
We all learned more languages than we started with
Made a functional product within just a few days
Used the power of AI to bring easy and fun translation to all

What we learned

Establish a UX flow early! We didn't have a functional project for nearly the whole hackathon because all of the developed pieces had no way to access them.

Matt learned Tasse, German for Cup. He's pleased about this.