RoboProf - Your new favourite virtual language coach

Menu Card displayed in Restaurant Setting
Menu Screen
Starting Environment
Learn Module in Restaurant Setting

Inspiration

In the past years, society has evolved in terms of technology and so has people’s need for growth and learning. Education itself has been redefined. As companies and governments open up to a hybrid economic and work model, it can be assumed that teaching/learning increasingly relies on a combination of personal and digital learning spaces. Due to the importance of languages from an individual perspective as well as from business and cultural perspectives, is it not important that language learning should also be incorporated into these virtual spaces?

What it does

Our project aims to combine the benefits of VR and NLP for an enhanced learning experience. We simulate common real-life environments that a person will have to face while travelling, like Airports, Hotels or Restaurants in virtual reality. In these situations, RoboProf, our robot professor, takes on various roles to teach and converse with the user. For example: in a restaurant, RoboProf takes on the role of a waiter and the user learns to enquire about various necessities like asking to be shown a table, requesting a menu card, giving their orders to the waiter, requesting details for a specific item on the menu, etc.

All this is taught to the user in two stages: Learn and Practice. In the Learn stage we teach the user the basics and allow them to progress through the stage using multiple-choice questions(MCQ) where they have to select the option that is most grammatically and semantically correct. The Practice stage is an open world environment where the user can interact with RoboProf. They must navigate the stage by chatting with the professor.

How we built it

The application is built to be used with a 6DOF VR headset. This is to enable positional tracking or room-scale tracking where the user’s real-life movements are mirrored in the virtual world for maximum immersion. Users may also reach out and interact with numerous items in the virtual world using their motion controllers.

We built the application using the Unity game engine and the AI part uses NLP techniques for implementation. We not only need to be able to respond to our users with appropriate answers but also, correct their grammar as and when necessary. To achieve the same we built a chatbot which uses the fastText library for word embedding and cosine similarity for operating the rest of the chatbot. On top of this, we use the BERT transformer to indicate whether the input sentence is grammatically incorrect or not. We also took the help of CockroachDB’s distributed database to store various data about our virtual environments. With the help of simple SQL statements passed with the help of python’s psycopg library we were able to communicate with our serverless cloud database.

We then integrate the Python code in Unity using Python.Net and an embedded version of Python. The C# scripts used in Unity to program the game behaviour make calls to the Python packages, send the input via these calls and then handle the outputs received.

Challenges we ran into

Trying to integrate Python and Unity in order to be able to use data science libraries like sklearn.
Finding a good database for situation specific chatbots.
Learning how to use CockroachDB for the first time.

Accomplishments that we're proud of

Being able to build a project like RoboProf which provides a practical platform for language learning was our primary motive and completing this was an achievement in itself. We were able to overcome the challenges that we faced as a team and worked together to make this project worthwhile. We also had a lot of fun along the way! :joy:

What we learned

We made a complete project in Unity and Python from scratch. A lot of the libraries and techniques we used were unknown to us prior to developing this project like the fastText library, Python.NET package and the serverless CockroachDB. Through this project we were able to work with these wonderful technologies and appreciate their utility.

What's next for RoboProf

Our current application only consists of a single restaurant environment. We aspire to expand our project to incorporate various other situations like airports, hotels, sightseeing locations, etc.
Our module so far is only able to indicate if the user input sentence is grammatically correct or not. In future, we can develop it further so we can make the algorithm display the grammatically correct version of the sentence if the user inputs an incorrect one.