heAR

Augmented Reality and NLP for Enhanced Communication

Comment

Inspiration

As first year university students, attending university lectures has been a frightening experience. Much different than how it is in high school, with a class of thousands of students, it’s much too easy to lose focus in lectures. One bad night of sleep can lead to valuable information lost and while one may think that only missing one lecture is alright, wait until they see the content they missed on the midterm. That is why we decided to build an application capable of speech to text recognition and a bunch of other useful features such as text summarization and text translation in order to help students with understanding lectures and avoid those days where one may zone out in class.

What it does

Paying attention is hard. Only 6% of the world are native english speakers. heAR uses AR and NLP to help you take notes and understand other people

  • To help the world understand each other better
  • To improve the education of students
  • To help connect the world
  • So you can focus on the details that matter
  • So people can talk about the details that matter
  • To facilitate deeper human connections
  • To facilitate understanding
  • To facilitate communication

How we built it

In order to build our project, we first came up with how we wanted our application to look like and what features we would like to implement. That discussion lead us on deciding that we wanted to add an Augmented Reality feature on our application because we felt like it would be more immersive and fun to see summarized notes that you take in AR. To build the UI/UX and augmented reality of the app, we used Unity and C#. In terms of text summarization and text translation, we used Co:here’s and Google Translate’s API in order to achieve this. Using python, we were able to build algorithms that would take in paragraphs and either translate them, summarize them or even both. We decided to add the translation feature because in University, and also in real life situations, not everyone speaks the same language and having that option to understand what people are saying in your own language is very beneficial.

Challenges we ran into

A huge challenge we encountered was having Unity interact with our Python algorithms. The problem we faced was that our product is on a mobile phone and running Python on such device is not really feasible and so we had to come up with a creative way to fix our situation. After some thought, we landed on the idea of creating a backend python server using Flask where our C# code would be able to make request to it and vice versa to retrieve the data we wanted. While the idea seemed very farfetched at first, we slowly tackled the problem by dividing up the work and eventually we were able to get the server running using Heruko.

Accomplishments that we're proud of

A huge accomplishment that we are very proud of is our working demo. The reason why is because in our demo, we have essentially achieved every goal that we have set during the beginning of the hackathon. From registering speech to text in Unity to having text summarization, we have accomplished so much as a team and are very proud of our finished demo. As the project went on, we obviously wanted to add more and more, but just having the feeling of accomplishing our original goals is truly something we will cherish as a team.

What we learned

We have learnt so much from building this project; from improving our existing skills to learning more, we understood what it is like working in a team environment. Not only that, but for all of us, this is either our very first hackathon or first hackathon in person and so we have truly experienced what a hackathon really is and have learnt so much from industry professionals.

What's next for heAR

To be honest, we are not really sure what is next for heAR. We did plan to add more UI/UX and Co:here features and possibly will continue or maybe venture into another topic.

Built With

Share this project:

Updates