We often see our friends on the spectrum who struggle with understanding emotions through facial expressions become flustered when they misinterpret the situation. It's not only an issue to identify the emotion, but also identifying how subtle or intense of an emotion the facial expressions is conveying.
How we built it
- Android Studio
- The UVCCamera library
- Microsoft's Azure Face API
How it works
The FeelReveal mobile application utilizes an external web cam to capture a video feed. The UVCCamera library was used to communicate with the usb web cam and capture frames from the video feed. those frames are sent to the Microsoft Azure Face API and interprets the average emotion of all faces present in the shot. The interpretation process involves drawing a precise rectangle around each face, then running machine learning models to determine how likely someone is to be feeling a certain emotion. The API returns an array of probabilities of each emotion through JSON, with the highest probability emotion being the most accurate. Our application causes the phone to vibrate in a specific pattern, corresponding to what emotion someone is feeling. For an added level of discreteness, a notification of the emotion is sent straight to your smartwatch as well!
Challenges we ran into
Google Cloud Platform's Emotion Detection is in beta - and is currently extremely inaccurate. We struggled for close to 4 hours testing the API, we eventually switched to Microsoft Azure. Due to the API limitations of Microsoft Azure, we are sending a frame every three seconds to be analyzed.
What we learned
This is everyone's first time working with the Microsoft Azure Face API, and it was a successful first attempt. Half of the team was new to Android Studio and never used the application before. We all learned and improved our mobile development abilities as a result.
What's next for FeelReveal
- Add voice recognition and sentiment analysis to enhance accuracy of emotional detection.
- Add facial recognition system to aid people who also suffer from face blindness.