The internet and current text based communication simply does not promote neurodiversity. People, especially children, with developmental disabilities such as autism have a great deal of difficulty recognizing the emotions of others whether it be verbal or written. The internet gave us the ability to communicate with each other easily. In the new wave of technology, we believe that all humans should be able to understand each other easily as well.

What it does

AllChat works like any other messaging application. However, on top of sending and receiving messages, when you receive a message it displays the emotion of the given text so that those with developmental disabilities can gain more insights and more easily understand other people's messages.

How we built it

The NLP system uses tensorflow and BERT to categorize text into 5 different emotions. BERT computes vector-space representations of natural language that’s suitable for use in deep learning models. The BERT family of models uses the Transformer encoder architecture to process each token of input text in the full context of all tokens before and after. BERT usually just classifies text as either negative or positive, so I had to fine tune it to get the model we have of classifying text in multiple categories.

Sockets were used to communicate between different IP addresses and ports.

Threading were used to stream text in and out at the same time.

The frontend system uses Kivy, a python front end library meant for cross-platform devices and multi-touch displays.

Challenges we ran into

There were a lot of firsts for this group. We are a bunch of first years after all. Whether it was someone's first time using BERT, or first time using kivy, there was a lot of pain in setting things up to a point where we were comfortable with the results. It was especially difficult to find good training data for BERT. It was also difficult to connect front-end to back-end with how the time difference in some of our group members was.

Accomplishments that we're proud of

For training the NLP system we had to read a lot of research papers about how labs have done similar things. It was extremely cool to apply something out of research papers into our own work.

All things considered the front-end system looks very good, considering none of us are designers and it was that members first time using kivy a lot of progress was made.

What we learned

A big lesson that continues to be relevant in the space of data science and machine learning is garbage in, garbage out. A model is only as good as the training data you provide it with. On top of that, we learnt to work better as a group despite our time difference by using github better and writing more meaningful commit messages.

What's next for AllChat

Some next steps would be to move to a server instead of having messages being analyzed on device as with long messages it can become time intensive for a mobile phone. On top of that, some security features such as end to end encryption would also be necessary.

Built With

Share this project: