Growing up, we have many deaf friends. Hongbo studied at Rochester Institute of Technology, one of the universities that admits the largest number of deaf and hard-of-hearing students. He often helped deaf students take notes in class. Chloe has friends who are partially deaf and hard-of-hearing due to heredity, illness, and accidents. She has helped them take notes in class and picked hearing aids. We noticed the challenges that deaf and hard-of-hearing students face and the limitations of the existing assistive hearing system at school and personal hearing aids in helping students fully comprehend the content in classes and focus on learning.

What it does

Our application aims to create a smooth, streamlined learning experience for deaf students in a classroom. It listens to the class, translates and transcribes audio to texts so that students can focus on learning and taking notes. We hope that deaf and hard-of-hearing students will never need to ask for help from classmates, come in class early to sit in the front row to hear better, or feel frustrated when struggling to hear and write at the same time.

How we built it

We use JavaScript, React, Redux, and redux-observable to build the front end. The front end is split into half. On the left, we use Google Speech to Text API to transcribe audio to text in real time with large, readable font for deaf students to understand the content of the class immediately. We also use Google translate to allow deaf students to select another language if their native tongue is not English. On the right, we use Google OAuth for authentication and authorization. Once log in, our app generates a Google Doc when students click "create note". We use GraphQL and Koa for the back end and for real-time communication.

  • Google Speech To Text
  • Google Translation
  • Google Docs
  • Google Drive
  • Google Storage
  • Google Client Authentication
  • React
  • Redux
  • redux-observable
  • RxJS
  • GraphQL
  • Koa
  • Bulma

Challenges we ran into

We have many ideas, features and functionalities that we want to build, but there is not enough time. Time management is crucial. This means we had to prioritize features that will allow us to have a minimum viable product and make a series of decisions on-demand.

In addition, having a rich but complex user interface can be distracting. Finding the balance between building a clean user interface and adding multiple features to our application while making it intuitive and easy to navigate is one of the most interesting challenges we faced.

Accomplishments that we're proud of

We are a small team, but we are proud of being able to integrate multiple Google Cloud APIs within such a short period of time. Above all, we take most pride in building an application that people will actually want to use.

What we learned

It takes good engineering capabilities to build a good application, but it takes much more than that to build a great application. During the hackathon, we brainstormed ideas, designed the user interface, decided on the best technologies to use, split the workload but also collaborated, changed part of our design on-demand to meet the technical and temporal limitations, completed the minimum viable product, and reflected on what we learned and what we could have done better for our target users. This has been a fun, challenging, but extremely satisfying and rewarding experience for us.

What's next for Third Ear

We would love to conduct user research to understand how our app can add functionalities that help our target audience learn better in class room. But we also believe that this app can serve more people than our target audience. For example, our native tongue is not English. This speech-to-text and translation functionalities can make it faster and easier for non-native students pick up new vocabulary in class.

We welcome suggestions and feedback. Let us know what you think!

Share this project: