Autism as we know of, is a developmental disorder which impairs social communication and interaction. Autistic children have trouble with emoting, unlike their otherwise normal peers. Education of autistic children is based on the subject of conditioning where the child is subjected to visuals repeated over a period of time, since the regular format of education has proved to be ineffective on them. In order to bridge the communication/social gap between the autistic child and rest of the world, our product 'EmotiLearn' records vital body parameter(pulse signal) and tracks the emotional changes as a physiological basis to it.

What it does

'EmotiLearn' is an assistive kit which consists of a wearable wrist band embedded with a pulse sensor along with assistive writing board. The wearable device contains a series of LEDs that provides visual based learning experience to enable the child to practice writing. This is followed up by a interactive gaming kit meant to teach the child to overcome social anxiety. The game is designed to provide a visually interactive platform, conditioning the child to repeated visuals to help adapt to the scenarios which are encountered on a daily basis. It starts off with a simple thought of having to meet a 'friend'. The idea is to make the child feel comfortable about performing such a task. To make the game more realistic, child's most favorite toy is used to start the game by placing it on the gaming console, this toy behaves as the main character of the game played. Once the child feels comfortable performing this, the next level involves meeting the friend to play, which again involves physical interaction between the favorite toy and another toy which acts as the friend. As the child is made accustomed to such scenarios the 'difficulty' level increases. The game console is integrated with the wearable wrist band for monitoring. The pulse sensor on board, records the pulse signals of the child as he engages in his routine activities as well as when he is playing with the gaming kit. This recorded signal is used to determine the current emotional state, in this case - happy or stressed. This aids understanding the state of mind, likes/dislikes, which collectively helps the caretaker provide better and adaptive preventive diagnosis. For example: in a scenario where the child is experiencing agitation, the caretaker is immediately made aware of the situation and is better prepared to intervene and prevent the outburst.

How we built it

The wearable device is build on arduino uno board, that acquires the raw signal data from the pulse sensor and commands the working of the LED based writing assist system by establishing serial communication with controller acting as server and Matlab the client. For pulse sensor, the raw data acquired from the controller is subjected to signal processing to remove unwanted noise signal. This processed signal is further subjected to feature extraction to determine the inter beat interval. A very critical factor in determining emotion state is the heart rate variability (HRV). Power spectral density of the pulse signal is estimated and from this we calculate the ratio of high frequency and lower frequency component to determine HRV. In our current implementation only a single level of classification is employed where we determine if the user is either happy or stressed (HRV<1 is considered happy). To achieve more accuracy in predicting different levels of emotion, we need to acquire more data sets from children with autism, it can be then trained and classified. The interactive game is developed to make the autistic child aware of routine social interactions and help build a positive mindset towards it. The console is built over a an ATMega32u4 controller running Arduino Leonardo firmware. It uses Human Interface Device (HID) protocol to communicate with the computer, and replicates the behavior of keyboard keys, mouse clicks etc. A simple game to prove the concept is built on a popular online software "Scratch".

Challenges we ran into

There is a dearth of information regarding the classification of emotion of person using the HRV. Also, choosing the right concept for the game development proved to be tricky.

Accomplishments that we're proud of

Ability to classify the emotion experienced by the person to a satisfactory level. Understanding the needs of an autistic child and narrowing down to a good concept for the game in a a short period of time.

What we learned

Signal processing in MatLab, study of physiological parameters, creating simple game using scratch.

What's next for EmotiLearn

Gathering more data sets to improve the accuracy to recognize emotion, creating a mobile application for monitoring purposes, develop games that are more adaptive to the change in the emotion, data gathered from this research can be used for other scientific utilities.

Built With

Share this project: