Live Speech to Text Overlay

Inspiration

Brayden started the hackathon with a story: while playing Valorant, a communication-based tactical shooter, one of his teammates sent a message in chat. That message asked everyone to type important communications in the chat, because the individual was deaf. We wanted to make a variety of activities easier for individuals like them.

What it does

The user's live audio will be recorded and transcribed to text sentence by sentence. Phrases are sequentially displayed to the screen, bottom up. Key phrases appear at the top of the overlay, allowing the user to see important information at a glance without reading every word. The primary use case we focused on is for hard of hearing individuals playing video games. For some video games, like Valorant, communication is a very important aspect of the gameplay--an aspect which is of course less accessible to hard of hearing individuals. We wanted to allow hard of hearing individuals the opportunity to play tactical, communication-based games without being at an unnecessary disadvantage. The key phrases include important communication information in Valorant, but in the future can be expanded to include anything the user finds important. This means the user will not have to read every word in order to quickly react to the situation they are in. The advantage of an overlay is that you can open new windows (even fullscreen windows) while the captioning stays on the screen. This is necessary for the program to run at the same time as something like Valorant.

Though we focused on one use case, this technology can be used for a variety of situations. If a user is in a call, for example, this technology will allow hard of hearing individuals to have a better experience and be more confident about what others are saying. This could also be used for individuals taking online courses to better hear their instructors. It could similarly be used for individuals who are taking job training online. Broadly, anything involving online audio communication can make use of technology like this to make life easier for many.

How we built it

We only used Python for this project. We used wx to create and change the overlay, and took advantage of others' audio recognition to update the overlay.

Challenges we ran into

By default the audio recognition software uses the user's microphone. We were initially concerned that this could not be changed to capture the audio of the computer (which would mean everyone on Brayden's team would have had to download the program, not just the hard of hearing individual).

Accomplishments that we're proud of

Some of us are new to coding, some of us were not familiar with Python, but we all worked together to build something we think is useful for the community.

What we learned

The power of friendship :)

What's next for Live Speech to Text Overlay

One major limitation is the time it takes for our program to give the user relevant information. Because quick communication is so important in games like Valorant (and in general), we would like to work on making this project run much faster. We would also like to expand the list of key phrases and allow the user to add their own key phrases to be shown at the top of the overlay.