This project started out as a simple idea: to display real-time subtitles for the hearing impaired in a convenient manner. Microsoft's HoloLens struck us as the perfect platform to achieve this. We then extended this concept to other settings - language translation, automated conversation scribe, context recognition - all of which revolve around the common thread of human verbal interaction. These fusions of ideas became our pride and honor, the HelloLens.

What it does

The HelloLens is a suite of features that assist people in various conversational settings. Our first feature, Real-Time Captioning, is primarily meant for the hearing impaired. Our second feature, Real-Time Language Translation, is of immense value to people from all walks of life. Our third feature, Chat Scribe, allows users to revisit past verbal interactions that they have had.

How we built it

We would like to thank Carnegie Mellon's Human Computer Interaction Institute for lending us the Microsoft HoloLens, a huge opportunity to create something that's truly beautiful and futuristic. Kudos to the Microsoft Azure team, who were so kind to help us as we learn the power of HoloLens and Azure. We are incredibly awed by the extent to which these two Azure APIs have made our work easier: Translator Speech, Speaker Recognizer. Credits to Firebase for powering our third feature, a way to recollect chats observed by the HoloLens on your mobile phone. We are so glad to have built not just a single app, but a mixed reality app and a mobile app, powered by our own original chatbot; not just a random functionality, but a series of related solutions to real problems in society.

Challenges we ran into

Any ambitious project will rattle under the enthusiasm of its team members and the complexity of the problems it is attempting to solve. Our biggest challenges lie in continuously improving our product despite the imperfectness of the state-of-the-art technologies that we have now. It has also been a challenging weekend picking up a lot of the necessary skills, the most prominent of which is developing on the HoloLens.

Accomplishments that we're proud of

We are proud to have built a product that can propagate social good in our community. We are glad to be leaving Waterloo having made many new friends and enjoying every minute of the 33 hours of hacking.

What we learned

Besides the hard skills that we learned, we had a fun time in another country, learning from the many wonderful hackers that occupy Waterloo's engineering buildings this weekend. We learned a little bit of project management, product design, and how to operate as positive members in a team.

What's next for HelloLens

We are considering doing a research project in wearable computing and real-time captioning at our institution. We will be exploring some of the features that we had no time to implement, including sentiment analysis in conversations. Ultimately, HelloLens is another wonderful step in building awesome products that mean a lot to us; since we have explored verbal communications, we look forward to tackling problems in other domains in the near future.

Share this project: