Virtualingo is an educational Virtual Reality experience born from a passion to teach language through immersion. Virtual Reality is a technology that can transport people across the world, and we sought to use this power for education. Our experience places the user in a Mexican diner, where they're responsible for taking orders in Spanish, building traditional Mexican cuisine using a variety of ingredients, and ultimately delivering a delicious meal to a satisfied customer. Virtualingo was designed to teach more than just language, but also culture and tradition. We want users to learn about various Spanish foods, traditions, and vocabulary.

We strongly believe that this technology could make a difference toward those trying to learn a new language, especially students in school that must do so as part of their curriculum. VR, in many ways, is the future of education, and can offer an immersive supplement to standard education. Gone are the days of relying solely on textbooks and tapes of people speaking foreign languages, and instead, VR has arrived to immerse users and let them explore culture without even leaving their rooms.

Our system takes advantage of IBM Watson to perform live translation, so in reality Virtualingo is prepared to teach in any language. We're excited to see where this project can go in the future as we integrate more languages, cultures, and immersive environments.

What it does

Virtualingo is a VR experience that teaches language through cultural immersion. For this demo, we specifically hoped to teach Spanish by having the user cook traditional Mexican cuisine and take orders for food in a foreign language. The experience starts in a kitchen, where the user is surrounded by various ingredients. There's a prompt in the room that displays what the customer wants to eat, but with one catch - it's in Spanish! Fear not, even if you don't speak Spanish (Which, hopefully, you don't, since this game is all about teaching you Spanish), you can explore the ingredients around you and grab them to read their Spanish name. That's right - Virtualingo helps you discover your hidden dream of becoming a Spanish chef who doesn't speak Spanish! Throughout the experience, you combine the various ingredients and follow the in-game prompts to create traditional and recognizable Mexican dishes, including tacos and tostadas. We hope that by the end of the experience, you can memorize the names of all these different foods in Spanish. Just in case you ever find yourself accidentally working at a restaurant in Mexico.

How we built it

Virtualingo is built for the HTC Vive, and was created with the Unity Game Engine, the C# programming language, IBM Watson, Autodesk Maya and Blender. We brought our own Vive to Calhacks (Trust us, finding space for a Vive at a hackathon isn't easy. Especially when fire hazards come into play. That's where the real engineering happened) and decided on our project from there. Many of the models, including all of the ingredients and the kitchen, were made and 3D modeled by our team. The language translation was all done through IBM Watson, since we live translate everything at runtime to ensure we can support multiple languages in the future. All functionality was programmed by our team as well.

Challenges we ran into

We constantly had to remind ourselves we were making an educational game about language and not a cooking game. You'd be surprised how easy it was to think we were remaking Cooking Mama for VR. Except we weren't. We were trying to teach a foreign language through Mexican cuisine. Those are two very different things. In many cases, we realized we would be focusing on the wrong priorities and working on tasks that really should've been saved for the end. Oh well, at least you can make some rockin' tacos. In Spanish.

In all seriousness, one challenge we did face involved making an active effort to avoid stereotypes. Our goal was to teach culture and language, and we recognized that by falling into common stereotypes, we'd be working against our end goal.

Time was also our enemy. We wanted to include speech in our project, also using IBM Watson, and we even got the basic functionality in our game, but simply didn't have time to implement it.

Accomplishments that we're proud of

On the bright side, we made both a cooking game and an experience that teaches language using Virtual Reality. We hope. It's a bit of both, and trust us, it was one heck of a rush. Also, we used IBM Watson to translate everything from English at runtime, with the goal of making our system extendable for any language in the future, which is pretty awesome. We also did get speech to text working, but didn't have time to implement it as we wanted.

What's next for Virtualingo

There's a lot of refining, refactoring, and improving to do. We'd like to improve the game's educational capability, while also polishing the cooking mechanics. This project was also initially supposed to have a second phase where the user must speak in order to take orders, and we had already begun using IBM Watson to accomplish this goal. In addition, we want to move beyond the sole medium of "cooking" to learn a language, and include other environments, situations, and practical applications where one may need to learn a foreign language. Food, and Spanish, is just the beginning. We designed our system to be extendable for any language, and we hope to include more of them in the future.

Share this project: