HoloSubs

Understand the World Around You

Comment

Inspiration

The idea of a real-time, VR translator came to us when one of our members went to Japan and was terribly clueless when it came to their language. Being too embarrassed to whip out his phone and make the store clerk repeat themselves, they wished that there was a device which translated their message on the go. And now there is :)))))

What it does

HoloSubs is a speech recognition and translation application built for HoloLens that will recognize the speech of the person the user is talking to and display the text and translate it in real time to the desired language set by the user. This application has the potential to make a difference for people who live in a foreign country that speaks a different language.

How we built it

Originally, we planned our app to be for Hololens. Unfortunately, the hardware never arrived so we decided to code the app for an Android Phone. We ended up using a camera to represent what you would physically see if you were to wear a HoloLens. The words would appear at the bottom to represent the “subtitles”. We used Locale to get the language and display it in a text view. Unfortunately, we were not able to get translations to work, but they would replace the “subtitles” at the bottom.

Challenges we ran into

One of the challenges that we ran into was that the physical hardware didn’t arrive at THacks. Due to this inability to code the Hololenses, we decided to compromise by emulating it through our phones. We coded an application that would make use of our phone’s camera, but the computer couldn’t recognize our camera.

Accomplishments that we're proud of

Due to the lack of hardware, we were not able to run our program with the HoloLens. However, we implemented a similar version onto android devices with the camera, speech recognition, and translation. Instead of viewing through the HoloLens, we used the phone camera to represent the user’s view.

What we learned

We learned how to render objects in 3D, and took a shot at creating virtual planes in augmented reality. Despite Unity being a new and overwhelming language us, we were able to learn critical concepts through this project as a team. We were also exposed to how VR software can communicate with the hardware and vice versa, which is a first for most of us.

What's next for HoloSubs

Having the program run on the actual HoloLens. We also want to have a real-time translation with any of the languages. Another feature we hope to add in the future is suggesting possible responses to demands/questions that other people ask.

Share this project:

Updates