SpeakEasy

Character Input
Phrase Selection
Confirm Phrase and Speak

Inspiration

The American Speech-Language-Hearing Association estimates that are over 5M individuals in the United States who could benefit from augmentative and alternative communication (AAC) technology while only an estimated 50k individuals use them. People with conditions such as ALS, Cerebral Palsy, Rett syndrome , etc. depend on these devices to communicate with family members and caretakers which makes solving these problems vital.

But why do so little people use AAC devices? Well we think that existing AAC/eye-gaze solutions are too expensive ($5k+), not expressive enough, and way too slow for real conversation. Traditional eye-gaze devices have an estimated range from 5-10 WPM compared to and average of 120 WPM for normal speech. Additionally existing devices struggle to allow users to include more fringe and personal words in their phrases, sometimes not letting them expand outside of a few preset phrases.

We wanted to fix these issues so that no matter your situation, you always have a meaningful voice.

What it does

SpeakEasy combines two methods to allow users to speak at much faster rates while also allowing for rich and detailed conversations. Users first type the initials of every word in the phrase they want to say using our PPM typing model. Then using an LLM + a long and short term memory system, SpeakEasy expands these initials to 6 candidate phrases. The user then selects the correct or closest phrase. If they like the phrase they picked they are able to say it with a cloned version of their voice. If they need to make some changes to the phrase they are able to pick individual words to modify easily swapping them out with other options.

The most exciting part of SpeakEasy is its memory system. Over time it learns who the user is, their preferences, and how they communicate and uses that to augment the candidate phrases. This allows for more niche words to easily be included in conversation.

How we built it

We first decided that for SpeakEasy to be fast our LLM and memory needed to be fast and accurate.

To do this we decided on building a long and short term memory system with Moss. The long term memory stores facts about the user between conversations allowing for more personal language. The short term memory stores the current conversation topic and context allowing for more accurate and relevant candidates. Moss was great for our use case due to the <10ms retrieval times.

Secondly we knew that for SpeakEasy to be truly expressive we needed an accurate voice cloning model that could mimic the users voice, or allow them to customize it if they are unable to speak. To do this we used Cartesia which provides fast yet accurate text-to-speech models. Additionally, Cartesia's speech to text allows for more context in conversations, making responses more relevant.

But the most important part of making SpeakEasy work was developing a UI that eye gaze users could actually work with. We referenced existing eye gaze software and adapted pieces of their technology to our system allowing for easy to use UI that can still reach high speeds. To reach these high speeds, we used a custom Prediction by Partial Matching model (PPM) to influences the sizes of each letter as they appear while typing. This helps letters that you are intuitively looking for larger than ones you likely aren't.

Challenges we ran into

The biggest challenge was getting the eye tracking to work. Unfortunately the one part of eye gaze systems that you can't afford to cheap out on is the eye tracker itself. We weren't able to spend $500 on a real eye tracker so the eye tracking implementation can be inaccurate occasionally. However SpeakEasy was not built to solve eye tracking but instead was developed to see how far we can push the limits of speed and expressiveness with eye gaze technology. In other words we built the UI system to push this limit.

Accomplishments that we're proud of

Developed a multi modal communication system that far exceeds the speeds and expressiveness of existing solutions.

With minimal user training we are pushing speeds of 25 WPM compared to an average of ~7 WPM for existing solutions.

Built an accessible UI that AAC users would actually be able to use complete with customizable accessibility options.

What we learned

We learned how to create a dynamic long and short term memory system that learns about its users. Additionally understanding what eye gaze users need from new AAC technology was incredibly important so put a lot of effort into that.