Autism, or autism spectrum disorder (ASD), is a lifelong developmental disorder that hinders a child’s ability to communicate and interact with others. Children with autism usually have trouble understanding nonverbal communication as well as back-and-forth conversations. It is estimated that 1 in 54 children has autism. Because autism is a spectrum disorder, the effects of autism are different for everyone and they can range from mild to severe. In the United States, autism is the fastest growing developmental disorder, with more children getting diagnosed with autism than AIDS, diabetes, and cancer combined. I have a personal connection and know several children with autism and I wanted to create something that would help them and their struggles with recognizing emotions. Current mobile apps to help children learn to recognize emotions use cartoon flashcards, a process that is boring for children. Additionally, cartoons cannot fully embody the complexity of human emotions. Thus, I wanted to create an app that allows children with autism to learn to recognize emotions with real human beings and people rather than cartoons. Another struggle that children with autism have is stress. Many studies have shown that people with autism have trouble coping with stress, leading to a variety of personal and motivational issues that can affect their education and social lives. I wanted to address these two problems that children with autism face (stress and empathy) through a mobile app.

What it does

My mobile application provides two main features: meditation and learning to recognize emotions from real facial expressions. In providing these features, my app also focuses on meeting the accessibility needs for children with autism as well as providing a clean and well presented user experience. For the meditation part, users are given a list of royalty free music to listen to. When the meditation music plays, bubbles slowly move around the screen while expanding and contracting to signify breathing in and breathing out. All the bubbles shrink while the child breathes in and expands when the child breathes out, calming the child. The second part of my application is emotion recognition. Children point the camera towards another person’s face and with facial recognition and emotion detection technology, the app is able to identify the person’s emotion as joy, surprise, anger, or sadness. Then the children can match the emotion with the choices on the screen. The application tells them if they chose the right option while giving them encouragement if they answer incorrectly. Additionally, the children’s statistics are saved into a database and displayed on the screen so that parents and teachers can track their progress. Once a child has at least an 80% accuracy rate, he or she can move on to the next level, which encompasses more complex emotions such as fear and disgust. Our app also has text-to-speech capabilities to tailor to the accessibility needs of children with autism.

How I built it

Expressions is an Android mobile app built in Java with Android Studio. In creating the front end, I used Sketch to create the UIs before any programming. I learned about material design and creating clean and intuitive mobile applications. For the backend of Expressions, I utilized several APIs such as the Microsoft Azure Cognitive Services API and the Camerakit API. Additionally, I used Google's Firebase database, cloud storage, and authentication for Expressions. I learned how to use many different objects in Android Studio such as the SoundPool object in order to use an audio file. I also used Adobe After Effects for the moving bubbles.

Challenges I ran into

Because Firebase was an extensive and essential part of Expressions, I ran into many problems regarding the backend of my app. This was my first time using the Cloud storage features and I couldn't figure out how to get the songs for meditation from the Firebase storage. Luckily, Firebase has really thorough documentation that was helpful for me to solve this problem. Another problem I had was using the camera. I first tried to use Android Studio's built in camera object but I found out that it was deprecated. This meant I had to look into using SurfaceView objects and binding a byte stream to the SurfaceView in order to have a video camera. However, when I was testing it with my emulator, the video always showed up sideways. I was really confused at first and I tried to rotate the SurfaceView 90 degrees in order to display it properly and I wasn't sure how to send a "sideways" video to the Microsoft Azure API. I ended up ditching the SurfaceView and using the CameraKit API. However, the video stream still showed up sideways. After using a real phone instead of an emulator, the video showed up normally. I googled into the problem and determined that it was the emulator's problem because my computer's orientation is horizontal so it made my emulator's orientation also horizontal. There was no fix to this problem and I had to keep testing my app using a real phone instead of an emulator because of the camera. The last major problem I ran into was using the Microsoft Azure Cognitive Services API because it was my first time using it. I wasn't sure if the requests in the API were going through and I solved this problem using print statements throughout my code.

Accomplishments that I'm proud of

I'm really proud that I finished the app with limited time. Originally, I had two other partners but they both ditched me. I wanted to finish this app this weekend and I'm glad I learned new things as well as had fun creating this app. I am proud that I integrated Microsoft Azure into my app, as it was my first time using its services and APIs.

What I learned

I learned a lot of backend development through Firebase and the Microsoft Azure Cognitive Services API. I also learned how to make an app that meets the accessibility needs of the users.

What's next for Expressions

I want to make a settings page where children can turn on and off the text-to-speech features, as not every child needs them.

My Discord username is: Rebecca#9548

Built With

Share this project: