Development in the mobile industry in recent years has revolutionized the way we as a society communicate on a daily basis with one another. However, there is still a long way to go, especially in terms of improving accessibility in the communication field. With Over-Hear, we take the first step in achieving this goal by enabling the people with hearing impairment to communicate in a general environment more efficiently because, well, we believe life is worth listening to.

What it does

Over-Hear is a mobile app that notifies the user when anyone in the vicinity calls their name out to talk to them. As soon as the app registers the users name being called out, it vibrates the smartphone and issues a notification too. The user can click on this notification to go to the app, where the app converts speech to text in real time for the hearing-impaired user to comprehend and reply to. All the user has to do is enter their preferred name when they use the app for the first time. It's simple. It's convenient. It's intuitive. It's accessible.

How we built it

The app was build ground-up using android studio with the help of Java for the backend and XML for the frontend. We used google's speech to text conversion API to convert speech from the people nearby to text that our user can comprehend in real time. Finally, we also implemented the concept of a "wake word" which would call upon google's API to do the conversion. This wake word in our app is also the name of the user. This wake word concept was implemented with the hep of the Porcupine API (Open source).

Challenges we ran into

Integrating different API's into android studio proved to a challenging task with a lot of debugging involved. Moreover, as none of us were experienced with android studio, using git also took some time and effort getting accustomed to. Implementing the "wake-word" to trigger our app's functionality proved to be the most challenging task. However, our constant determination to get it to work kept us going through the night to solve the issue!

Accomplishments that we're proud of

A working project! Building something from scratch in a short span is never easy nor straightforward. Moreover, our project does more than just "work." Over-Hear is an idea than does use the best of machine learning and AI by putting it to a cause that helps the disabled in the society and we are extremely proud of that at "Over-Hear" (get the pun?!)

What we learned

Since none of us were veterans in android studio, we had to learn it on the fly. We also learned the importance of raw data which is used by our wake word engine "Porcupine." Ideating and implementing different ideas also proved to be successful and helped us expand our vision for the app. Moreover, we learned the criticality of adequate communication and collaboration between team members which lays the foundation for a great project.

What's next for Over-Hear

  1. Adding user-customizable names as wake-words to our app. The Porcupine API we currently use is a limited, open source version, with the full version requiring a license.
  2. We would like to add the functionality of storing conversation messages on the app for the users to review at any convenient time. This could be done with the help of google's firebase.
  3. Update real time speech to text conversion to support multiple languages to enhance the accessibility.
  4. Furthermore, we aim to publish our app to multiple mobile platforms, including iOS.
Share this project: