TuneIn

Inspiration

We often take for granted gaming with friends; this is especially the case during a pandemic. Most popular games like Among Us, League of Legends, and Fortnite are played on calls with groups of friends. For games requiring full engagement, chat options are often pushed to the side. Consequently, people with hearing impairments are left behind in one of the most popular ways to socialize.

We sought to change this.

https://www.reddit.com/r/VALORANT/comments/ih4pvw/in_plat_solo_queue_i_had_a_teammate_who_was_deaf/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

What it does

TuneIn is a chrome extension that loads pre-trained speech-to-text TensorFlow models to run in-browser and, using a stream-based approach, will produce system-wide notifications. The goal is to inform a player with a hearing disability on what their fellow players are saying. In order to be able to focus, the notifications should be curt and focus exclusively on game keywords. TuneIn currently supports League of Legends conversations with words like "ping" and "spawn".

How we built it

We trained a TensorFlow model to classify ~1 sec snippets of audio into one of our predetermined "dictionaries" of gaming jargon. Moreover, we developed a chrome extension that loads this model from the cloud and, in-browser, parses spectrograms of mic input into classified tokens. Should a token be unrecognized or background noise, we do not send out a notification.

Challenges we ran into

We initially were concerned with the speed of model predictions and/or loading given that the notifications are time-sensitive in relevance (especially for fast-paced games). As a result, we planned to use transfer learning to modify a pre-trained model from TensorFlow js (speech-command). The approach we took was to construct a light-weight/shallow neural network that we manually trained on a couple keywords. This network would then be appended to the pre-trained model. However, the accuracy of the model was poor and hyper parameter tuning proved to be extremely time consuming. As a result, we opted for a heavier, but more robust TensorFlow model trained using Teachable Machine.

Accomplishments that we're proud of

We're proud of how our extension is non-intrusive to the gaming experience, and seeks to connect people.

What we learned

ML is hard :/

What's next for TuneIn

We plan to allow for a more robust approach to how we support different games. Currently, adding support for a new game would require training a separate network and uploading it to the cloud. Alternatively, we could have a single network that does speech to text conversion and then use text summarizers like Pegasus to extract crucial information.