Chinmayee has been wanting to work on a similar project at University for some time, so when we came to BrickHack and saw the abundance of sign language users, we figured it was a perfect place to actually do it!
What it does
The AR app built for the Vuzix glasses has two functions. On the main screen, it can detect sign language and display it's guess to what is being signed on the screen. When swiping to the next screen, a user can speak with another glasses wearer to display a live translation of what is being said.
How I built it
We built this project using the Vuzix glasses, communicating with a Tensorflow model, along with Google Compute Engine and the Translate API.
Challenges I ran into
We ran into a few challenges, the version of the glasses we were using do not have Google Play Services support, so we could directly use the Translate API, or Firebase. We also ran into network issues when trying to stream video data.
Accomplishments that I'm proud of
Everything! The project, the demo, and the team <3.
What I learned
We all learned a lot, whether it was regarding machine learning, AR development, or websocket work :).
What's next for Babel AR
We plan to continue this at University with a 3d camera and better training data!