Inspiration
Children and teenagers with autism spectrum disorder (ASD) tend to have a hard time accurately interpreting emotions. For example, an autistic child might characterize all negative or unpleasant emotions as anger or they might not recognize when they’re excited. They are still able to experience a range of emotions, but need support to recognize, understand and manage them. This is a pressing issue because it can lead to difficulty in interaction with other people, restricted interests as well as symptoms that affect their ability to function in school and other areas of life. Studies have shown that Behavioral approaches have the most evidence for helping such individuals. With the proper support to recognize, interpret, and respond appropriately to other people’s emotions, autistic individuals can greatly improve their emotional interpretation. However, current solutions all include heavy reliance on caregivers, carrying out exercises such as Playing emotion cards (cards that have pictures of faces, either real or cartoon, which you can use to teach the child basic emotions). While effective, in a society with widening inequality, the intensive commitment of a caregiver that is needed means many autistic individuals may not receive the necessary support. This introduces a serious need for a simple way for autistic individuals to strengthen associations between emotions and various stimuli, without the need for a caregiver. This is where Colormotion comes into play.
What it does
Colormotion is an accessible resource that allows autistic individuals to strengthen associations between emotions and various stimuli on their own.
In order to ensure that our users are able to benefit from our Colormotion in all circumstances and scenarios, our product consists of a chrome extension and a cross platform mobile app (both Android and IOS).
Colormotion’s chrome extension highlights the background on the live-captioning feature on youtube to convey the overall emotion being portrayed in a phrase. Videos are powerful tools for teaching emotional responses. This already positive effect is significantly strengthened by our product. With the average child spending upwards of an hour on Youtube per day, this is a useful integration of our application. Using our chrome extension, this one hour can now simultaneously become a seamless learning experience, making it sustainable in the long run.
Colormotion’s app allows for live sentiment analysis through the user's microphone, displaying specific colors and emojis to match the overall emotion of the phrase recorded. The original motivation for this was in cases where individuals are processing information from a different medium, for example a TV screen where we are unable to modify the subtitles. However, with the flexibility of a mobile application, the potential use cases span much greater.
Both products map phrases to one of 6 different emotions: neutral (gray), joy (yellow), surprise (orange), sadness (blue), anger (red), and fear (purple).
Our product exploits the human tendency to associate emotions with color. By processing the color and the stimuli at the same time, autistic individuals can learn the relationships between emotions without being fully aware that they are doing so.
The ease of use and accessibility of our product means that individuals will easily be able to benefit from it regularly, which over time provides a way to strengthen their intuitive understanding of emotions.
By performing sentiment analysis live and displaying the appropriate emoji and color, autistic individuals can quickly identify which emotion is being expressed by others, making communication easier and more effective.
How we built it
App:
The mobile application was built with React Native, in order to ensure that both Android and IOS users are able to benefit from Colormotion. Our React Native application effectively utilizes an array of advanced React programming hooks and concepts. In order to allow for our required speech-to-text functionality, our application integrates native code, to access the device’s microphone. The React Native front makes POST requests to a web-hosted API for NLP results. To accommodate for this, on the front end asynchronous programming methods were utilized to ensure smooth execution without any errors.
NLP and ML Component:
For the most accurate estimate of emotion being conveyed in a given phrase, we experimented with several NLP and Voice Classification models. By representing each word as a vector through the word2vec model, we also attempted to see if the results would be more accurate if we weighted and combined the suggested outputs of both an NLP and Voice Classification model in order to account for the complexities often conveyed in everyday expressions. Ultimately we utilized a model trained on emotion labels for texts from Twitter, Reddit, and TV dialogues. We believe this is optimal, considering the contents of those media are similar to what we anticipate Colormotion to be used on. The expected accuracy of 66% is strong compared to the baseline of 14% when such emotion predictions are done at random.
Chrome Extension:
The chrome extension was built with vanilla JS with HTML and CSS formatting. When turned on, the extension scrapes and stores the subtitles on youtube from the local device. Once a sentence is complete, using the ML model previously described. After receiving the result of a corresponding API call the program highlights the background of the subtitles with the appropriate color. The extension also features an intuitive UI design that displays each emotion with the associated color and expression.
Challenges we ran into
We faced many challenges when creating Colormotion. The largest one being understanding the best way to improve emotional interpretation in autistic individuals. We experimented with facial tracking in videos to label people with certain emotions, an emotion guessing card game, but ultimately decided our final solution was the most effective.
Accomplishments that we're proud of
We are proud that we were able to build a fully functioning product that essentially functions on Web, Android and IOS. Both the chrome extension and app work hand in hand to create an effective way to strengthen emotional understanding in autistic individuals.
What we learned
On the technical side, we truly enjoyed working on this project and were able to strengthen our skills by working on this real world application.
Simultaneously, we learned a lot about autism spectrum disorder, and the intricacies of living with this condition. The difficulties that affected children and teenagers face motivated us to complete this product to the best of our ability in an attempt to alleviate some of their struggles.
What's next for Colormotion
With psychologists estimating there to be at least 25 different emotions, we hope to expand the range of emotions we are able to map to. Additionally, we also hope to further improve the ML model which our product relies on in order to refine the user experience on Colormotion.
Log in or sign up for Devpost to join the conversation.