-
-
Wandering around the VR environment
-
Picking up a VR object
-
I've always wondered how to say this in German
Inspiration
Both of us love learning languages, and wanted to do a project pertaining to the process of language-learning. Virtual Reality is an incredibly new and interesting technology that has a lot of potential in various fields. Immersing yourself in a language is the fastest and most effective way to learn it, and with VR you don't need to get a plane ticket to get access to such an experience.
What it does
Our project is a language-learning supplement designed for additional practice and immersion outside of the classroom. The user wanders through a scene littered with various every day objects, especially those tailored to the given subject or chapter (i.e. a school). When the user walks up to said items, it will give them the name of the item in english, and when they pick it up, it utilizes the old Google Translate API to give the name of the item in whatever the target language is.
How we built it
The tool was built in Unity 2018.3, and coding was done in C#. For the Google Translate API, we found an open source online resource that allowed for integration into Unity. In addition, we utilized the SteamVR Toolkit for the majority of the VR functionalities, including the character, hand movements, interactions, and teleportation.
Challenges we ran into
Our initial idea called for using The Google Cloud Translate, Speech-to-Text, and Text-to-Speech APIs. However, after about 4 hours straight of trial and error with integration, we discovered that the API is not readily compatible with Unity's infrastructure, and as such, we had to scrap using all of those APIs in favor of the deprecated Google Translate API, which isn't up to date, but still does a pretty good job.
The other issue with the above is simply a matter of scope. The end goal we were hoping for was a variety of immersive dialogs in a multiplayer environment, similar to the game VR Chat. We were hoping to set up servers for people learning a language at a specified skill level to converse with others from around the world with that skill level. We would utilize text-to-speech and speech-to-text to give and receive translation requests if you don't know how to say a word. For instance, saying "Okay Google, how do I say 'How are you' in Hindi?', and it responding with both the text displayed in a HUD, as well as the pronunciation. This, evidently, was vastly out of scope for a 24 hour period, especially with the above issues with the APIs.
Accomplishments that we're proud of
This was both of our first attempts at building a VR system from the ground up, and we are incredibly happy with the results. Although we didn't get all of the features we wanted in, it does the core of what we initially intended to do, which was providing an immersive language-learning experience.
What we learned
Although we didn't manage to get it implemented, we did learn the steps necessary to integrate with Google Cloud API, and as such, will be able to utilize those tools in future projects.
We also learned how to set up a VR environment using SteamVR, as well as utilizing HTTP requests to interface with an online API.
What's next for EducationVR
As mentioned above, we really hope to expand this experience to a more communal, discourse-based product. Dialog is the number one way to practice, and doing so in such an immersive environment could prove to have substantial benefits.
We also noticed that our implementation as it currently stands is limited in that it only allows for the learning and practicing of nouns. In the future, we hope to expand this to all the basic grammar types, as well as expanding it to full-blown lesson plans for professors teaching courses in a language to utilize. Such lesson plans would mirror textbooks, which already set role-playing scenarios for students to act out, such as going to a movie, boarding a train, shopping, etc. These types of lessons would be far more effective if the simulation was more immersive, which is exactly what our product provides.
Log in or sign up for Devpost to join the conversation.