Phoneme Based Augmentative Alternative Communication Device

Inspiration

Both teammates are disabled and passionate about increasing accessibility for other disabled people. AAC devices are one device that is helpful for individuals with difficulty speaking. However, for individuals who never had the ability to speak, it can be very difficult to learn how to read and write. That means they are limited entirely to pictogram based AAC devices which limits the amount of vocabulary they can access and take a long time to produce words with, even by the most skilled users. Studies have shown that phoneme based AAC devices might be a viable alternative for individuals with low literacy who need AAC. There is not yet a full phoneme based AAC device available that also incorporates predictive text features which allow for speedier entry.

What it does

The phoneme based AAC device lets people who are illiterate and have speech impairments to have conversations. It has keys with images to represent phoneme, at the bottom of each key is the word in which the picture is. On the word the phoneme part of the word is underlined. When the user clicks a key then there will be predictions of the next phoneme to be clicked on the left of the keyboard.

How we built it

The machine learning model was built with keras with databricks mlflow being used to save models as progress was made with building the application, allowing for a faster developing time. The front end was built with html and javascript with Flask being used to bring the frontend and backends together. ChatGPT was used to help speed along the development process.

Challenges we ran into

Some challenges we faced were that the predictions wouldn't show up on the web page when we tested it. Once the predictions appeared on the page they still needed to go onto the prediction buttons. We were able to solve this issue near the end of the hacking time.