As Generation 0 and Generation 1 immigrants ourselves, we all come from multilingual backgrounds. English fluency is a critical skill for success and assimilation in our society, we wanted to use some of the most exciting trends in industry - AR and IoT - in this space for maximum impact.
What it does
HoloComm is a real time translator. Users enjoy the following features:
- Visual output (on Hololens) of surrounding English audio translated into user's preferred language
- Visual output (on Hololens) of English text in field of view translated into user's preferred language
- Object recognition and labeling in English
HoloComm is also able to interact with our IoT avatar, Pikachu. The object recognition feature recognizes Pikachu as the avatar and the the user is able to initiate conversation. Pikachu, empowered by api.ai's natural language processing framework responds in conversational English. The entire conversation is tracked by the user using the same visual output on the Hololens as before and is transcribed to the user's email using SparkPost. Pikachu is meant to serve as an educational aide for users to practice their English at leisure.
How we built it
Lots of love <3 Many different technologies come into play here. The majority of code is in C# to interface with the Hololens. connecting to hardware occurs over a Firebase database which triggers Python running on a Arduino.
Challenges we ran into
Storing and processing snapshots of the visual field of Hololens to perform OCR and Object Recognition. Currently, the images are being pushed up to imgur with Microsoft Cognitive Services analysis subsequently.
Accomplishments that we're proud of
Building a complete user experience with rich interactivity.
What we learned
How to build a user interface in Unity for the Hololens. How to develop natively in C# for functionality usually accomplished in Node.js.
What's next for HoloComm
More features! An immediate addition is a gamified extension to object recognition which asks the user to identify certain objects with Pikachu as prompter.