Tempo

App interface screenshot
Wristband Prototype Design
Wristband Prototype Sectional View

Inspiration

The ability to speak smoothly and annunciate words clearly is an ability that we take for granted every day. For others this natural aspect of life may be much more difficult that we can see at first. We went to high school with a Deaf and Hard of Hearing (DHH) program; many of our friends back home are hearing impared. It has always been very inspiring to see how they've adapted to their situation, all with unwavering persistence. This app was created to help them, and to inspire them to keep succeeding and pushing forward every day.

What it does

Tempo is designed to help the user regulate their speech pacing; as you speak the app recognizes it, calculates the pace of the speech, and lets you know if the pacing of your speech strays away from more optimal speaking rates. The application recognizes the user's input speech and its pacing, and returns haptic feedback if the speech rate that the app reads is too quick on a scale. The faster your speech gets, the more intense the feedback.

As you speak, you can see the words you speak appear on the app interface in real time, as well as a visual that moves faster as you speak faster. If you start to speak faster, you will also notice the wristband system begin to vibrate faster as well, ramping up in intensity as you speak faster as well.

How we built it

Android Studio (Java), Arduino (C++), Dassault SolidWorks (Wristband Prototyping), Google Cloud Platform (Voice Recognition)

Tempo harnesses Google's Speech-to-Text service to convert user speech into quantifiable data. The app then measures the time in between words and uses a proprietary algorithm to return a rate of speech in real time. This entire application was designed using Android Studio and various APIs for interface animations and Google Voice Recognition services. The app then communicates to the Arduino through Bluetooth.

The Arduino was programmed to recieve a scaled speech rate from 0 - 10 from the app and then turns a servo motor back and forth with varying levels of intensity based off of the speech rate.

Challenges we ran into

The biggest issue we ran into was the intuitiveness and usability of the app. Tempo was designed to be used by the hearing impaired, and prioritizing their neesd throughout the design process was ambiguous at times. Many times we had to stop, sit back, and ask ourselves if this feature or this implementation actually made sense for the users that we were trying to help. We even consulted one of our hearing impaired friends in order to get a better outlook on what life is like for them, and how we should tailor our app to fit that. With these new considerations in mind, we were able to make many tweaks to best outfit the app for the hearing impaired, like emphasizing colourful and direct visuals and emphasizing feedback that does not involve hearing.

An issue we ran into was displaying the speech to text in real time as initially, we were only able to receive the result after we finished talking. However, with a significant aspect of the design being the ability to provide real time feedback to the user, we were able to display interim results as the user talked. Through looking at sample code and API documentation a little more closely, we were able to figure out a solution. Calculating the difference in time between words allowed for the user's speech pace to be determined and the appropriate feedback to be provided.

The serial connection with the Arduino also presented its own bugs and problems. The most prevalent one was how the Arduino actually recieved input data from the serial connection to the phone. It took us a while to realize that it actually received data in a strange form involving ASCII numbers, so once we realized this and took the appropriate measures to remedy it the Arduino got up and running.

Accomplishments that we're proud of

We are most proud of the fact that we can create something that could potentially change the lives of someone with a disability. The best quality of life is a goal that everyone, hearing impaired or not, would like to achieve, and we are proud that this project could be useful or even instrumental in doing so. While creating an Android application essentially from scratch and harnessing the Google Cloud Voice Recognition software is really cool, the greatest accomplishment is being able to use this technology to help people all around us. There are so many things that the hearing impaired have to worry about on top of everyday worries that we all have, and our only hope for Tempo is that it may serve to help allievate one of these worries.

What we learned

The most significant knowledge we gained during this design journey was a look into what life may be like for the hearing impaired. While we will never be able to experience what they go through every day, this app has shown us just how difficult speaking, something that we take for granted every day, can be for the hearing impaired. This struggle is exactly why we decided to take on this challenge.

And what a challenge it was. From incorporating all aspects of a fully functional Android application to reading miniscule pins and their documentation for Arduino modules, we learned that developing an application is much harder than it looks, and that there is much more to the thing that you can open with just a tap of your finger on a screen.

We also got a great look into many of the services and API's that Google offers, especially the Speech-to-Text and Voice Recognition Components of Google Cloud. We learned many incorporation and performance streamlining techniques for Android app development, as well as integration with assorted hardware and Arduino incorporation as well.

What's next for Tempo

There are so many places we can go with this app. On the horizon would be expanding to more platforms, like iOS and maybe even a PC/Mac app. We'd also like to go even further with the hardware, eventually including smartwatch, fitness watch, and other smart technology integration.