Two of our members' close family members have low functioning autism, which case them to speak rarely or speak in a non-sequitur incoherent sentences. Specifically, when asking them questions, they will usually respond with only one or two word responses that do not match the context of the conversation. Other times they were an attempt at answer the question that was impeded with a lack of language skill.
Because these individuals with special needs are unable to communicate their needs, they rely on their caretakers to translate their thoughts.Because special needs children have such expensive therapy that is often not supported by government funds, special needs families are financially burdened to send their children to speech therapy on top of the other private therapy and classes.
In today's world where soft skills are more necessary than ever, we wanted to use modern natural language processing (NLP) techniques to help children with language processing issues. To make speech therapy widely accessible and affordable, we wanted to teach children to communicate in full coherent sentences and gain confidence in themselves in their language skills so they require less reliance on their caretakers.
What it does
The program comes with a set of open-ended questions that are designed to let the user talk about themselves, as well as example answers. When the program is running, the computer asks a question to the user, and the user should speak into the computer.
The computer then uses NLP-based methods to check for proper grammar, as well as relevancy to the question. If the user says something that is irrelevant or non-sequitur, the computer will prompt the user to answer again in an attempt to correct the user. For instance, if the user is prompted by "How are you today?" and the user responds with "I like monkeys", the computer will respond with a correction. However, if the user says something that is relevant but does not have proper grammar, the computer will attempt to create an improved sentence that uses the example answers as a template and substitutes in the new words, which the computer will them prompt the user to repeat. When the user repeats the proper sentence, the computer will indicate a success, and moves on to the next question.
How we built it
We used Python as the foundation for this project. For both text-to-speech and speech recognition, we used Google Cloud's libraries, which integrated well with Python. To grammar check, we used linkgrammar, an implementation of the Sleator/Temperley/Lafferty theory of natural language parsing of link grammar, where a sentence is constructed through relations (links) between different words. To check relevancy of the answer to the question, we used NLTK's Word2Vec as well as Sent2Vec through vector analysis which indicates the relevance from a scale from -1 to 1. If the user does not meet the relevance threshold, it will ask the user to try again. Finally, in order to do intelligent substitution, we used the Spacy Module to detect parts of speech to substitute and synonyms to suggest a intelligible sentence. Both LinkGrammar and Sent2Vec use sophisticated machine learning methods to detect coherence in sentences and similarity between sentences respectively.
Challenges we ran into
Sorting out the dependencies between all of the different modules that we used was probably the biggest challenge we faced. Many of the essential libraries that we needed (google cloud, sent2vec, linkgrammar) were developed for different platforms or different versions of python, so getting them to all coordinate and integrate with each other took up a significant portion of our time. Furthermore, with the differences in our skillsets, we wanted to designate the work in an advantageous way while still letting each member of the team learn something new. Finding the right balance between optimizing for the project and optimizing for our own learning and self-improvement was another task that was far more challenging than initially expected.
Accomplishments that we're proud of
Helping special needs children with cutting edge technologies has made all of us extremely proud of the project that we created. Many of our team members have experienced firsthand the just how much of a struggle raising a child that learns differently from others is, and so the motivations for this project were very personal.
Completing this project was an incredible personal accomplishment, as we would be able to help those who are close to us who currently struggle immensely with regular teaching methods increasing the accessibility of our project.
Furthermore, on a grander scale, all of us recognize that special needs education is an area that many people overlook when discussing education in general. Thus, we hope that this project can help bring more attention to special needs education and the resources that those with learning difficulties require.
What we learned
This project required a deep understanding of linguistics that we often overlook. We learned computational linguistic tactics such as how to tokenize, stem, and lematize words to manipulate them properly.
This project prompted us to think about our thinking process when it comes to speaking in order for us to replicate an accurate AI. The thought in which individuals with special needs must go through to create coherent sentences is something we often take for granted.
We learned to develop a user-centric product that is tailored to their specific needs, we learned to put ourselves in their position to adapt to our audience.To identify areas our users might struggle in, we drafted flow charts to individualize each step of our thought process. Subsequently, we broke up our words into decision trees to formulate proper syntax and diction.
What's next for Speech Supporter
- Improving the performance of the substituting algorithm
- Generating more data from users to develop a more accurate machine learning pipeline†