Lingua Place

How many objects in your room can you name in a different language? Lingua Place transforms your room into a learning hub.

Comment 2

Inspiration

Our goal was to make language learning accessible and enjoyable for everyone, regardless of age or ability. Imagine being surrounded by everyday objects in your room and suddenly being able to name them in multiple languages. That was the inspiration behind Lingua Place. We wanted to create a fun and immersive way to learn languages using the objects we interact with every day.

What it does

Leveraging Mixed Reality and Google Translate API, you can learn multiple languages (English, Spanish, French, Hungarian, and Bulgarian) starting from the objects in your room. Master pronunciation, expand your vocabulary indefinitely, and reinforce learning through touch.

How we built it

This Mixed Reality Application was built using the Unity game engine, utilizing Meta XR SDK for Hand Tracking, Passthrough, Meta's Voice and Audio SDK for Text to Speech and Speech to Text recognition, and the Mixed Reality Utility Kit (MRUK) for scene understanding to label elements. Used Google Translate API for precise translations and Google Speech to Text and Text to Speech to expand the supported languages. OpenAI APIs were used to expand the application's vocabulary by connected words functionality.

Challenges we ran into

Currently Meta does not allow to access camera, and SceneScript is still in the process - this fact limited us in recognizing certain objects only. We overcome it by introducing the connected words, that can be refreshed on-demand - this allows extended possibilities.

Accomplishments that we're proud of

We're excited to provide a novel way to improve language learning. WIT.AI and Google API provides reliable and precise translations real-time. Once that SceneScript will be available, this made our app ready to translate any object around the user. We talked about accessibility, that's why we enabled pronunciation of words by touching the object and also gesture detection to activate the speech recognition function.

What we learned

We learned how to use MRUK and the interaction SDK, and how to integrate Google Text-To-Speech, Google Speech-To-Text, Wit.AI, and OpenAI web APIs.

What's next for Lingua Place

We will extend the AI generation of connected words to other categories like Synonyms, Adverbs, Verbs, as well as example sentences. The scheduled launch of Meta’s SceneScript will greatly increase the scope of words presented and truly unlock the potential of the application.

+ 1 more
Share this project:

Updates