Moodzy-App

Inspiration

We were inspired to create this app after reading various papers related with emotion recognition in conversation (ERC) such as Li, Lin, Fu, Si & Wang (2020). Although such researchers have made further advances than what we created, we wanted to take a leap into ERC ourselves to be able to one day add to current research on the topic as these topics are close to both of our heart as a psychologist who has worked closely with individuals on the autism spectrum and a biomedical engineer who has worked in the medical field developing medical applications.

What it does

The app essentially detects speech to turn it into text which then detects the emotion transmitted from said text. The speech recognition is directly made on the mobile phone, that text is then sent through the Modzy API for its further processing. The Modzy Sentiment Analysis model makes the mood detection, which the application will display.

How we built it

The application has been built using Ionic's Framewoork, allowing it to be a multiplatform app. The speech recognition uses the Ionic's built-in feature for that purpose which is a Cordova plugin. We also used amCharts5 for the app graph and superagent for the API calls along with some other open source Cordova plugins.

Challenges we ran into

One of the challenges we ran into involved the reception of the results once the text had been processed. The time that the Moodzy API takes for processing inputs is different each time, we had to create a pool that stored the identifier of every input sent, so in case that the results were not processed immediatly we could ask for those results later without loosing any input data.

We also spent some time trying to make the Moodzy JavaScript SDK work for TypeScript so we could use it with Ionic, we were not able to do it so we ended up just using supearagent the same way it is used in the Modzy API examples for both sending and receiving data.

Accomplishments that we're proud of

We have been able to process text in order to extract information from a mobile phone in a way that would have been very difficult to do without a computer. There is no data loss when a sentece is recognized and we believe that the results are self explanatory with the simple interface we created.

We have made the ground floor of what could be a much bigger project, with a lot of possible improvements to implement using the same technology and working for both mobile phone OS.

What we learned

We learned that thanks to the Modzy technology there is a lot of new open doors for mobile applications to grow, using neural models that usually consume a lot of the resources of a computer from a mobile phone could change the way a lot of applications are developed.

We also learned that there is still a lot of work to do within the AI emotion recognition field, there are not many applications and the possibilities are endless in a topic that has yet a lot to grow with a technology that we feel that is ahead of the projects done with it.

What's next for Moodzy-App

Our short term improvements include:

In the near future we look forward to being able to send small audio records over short periods of time so that an uploaded model to the platform can do its own speech recognition, this way no third party libraries will have to be used and the app can gain more independence and its response time should shorten with better accuracy.

Furthermore, we aim to be able to to send waveform images of the audio detected to the app which not only would produce better results, but would also be able to read emotions more accurately as it would add additional detail from the detected speech.

We aim to be able to further develop the app to a point where the app can detect complex specific emotions such as "sarcasm" from the tone recognition so that it becomes more accurate and can further positively change many aspects of our daily lives. For instance, being able to detect the overall mood in a conversation a group is having or detect the mood in a room, calculating the average mood from each conversation detected and producing a result. This may be a long shot for us considering the progress of our work however, we are not the only ones interested in researching this topic as many other great curious professionals out there are in the same position as us, dedicating their time to expanding this topic.

There is an apk of the project inside the github repository along with the code which can directly be installed using our own ApiKey. The github Readme also contains instructions to build the application directly from the source code

github