Sense+

Sense+ makes the approach to helping those in need proactive compared to the traditional reactive approach. It utilises speech, facial recognition and other technologies to infer emotions of users.

Comment 20

Inspiration

The global pandemic has revealed the growing issue and importance of mental health, in particular one’s accessibility to mental health services and the detection of someone suffering from stress, anxiety or other mental health conditions.

We personally have seen that being mentally well allows us ability to work and study productively.

It is the on going issue of those mentally unwell not approaching anyone due to societal stigma of seeking treatment that worries us.

Our project/proof of concept aims to make the change the approach of helping those in need proactive, rather than waiting for individuals to come forward by themselves, all whilst aiding to reducing the stigma associated with suffering from mental health issues

What it does

Our program integrates voice and facial recognition to detect/infer an individual’s emotions.

The voice using sentiment analysis to detect keywords from an audio transcript. These keywords are categorised as neutral, positive or negative. Natural language processing and regular expressions are utilised to break down audio transcripts into multiple sentences/segments.

The facial recognition uses convolutional neural networks to pick up features of ones faces, to identify emotions. Videos broken down into multiple frames which are fed into neutral network to make the predication.

This model is trained and validated using Facial Expression Recognition data from Kaggle (2013).

As of now we have nearly turned the above concept into an app which allows users to upload multiple videos, which are then analysed and results/predictions are returned about the emotional state of an individual.

The implications of this is that it can aid in indicating whether the user should seek professional help, or at the very least make them possibly aware of their current mental state.

How we built it

The frontend was developed using Java (Android Studio), whilst our backend was developed in Python, with the help of python packages such as TensorFlow, Keras and speech recognition. The frontend and backend communicate through Amazon AWS platform. AWS lambda is utilised so our code can be ran serverless and asynchronously. S3 is employed as a bucket to upload videos from the frontend so the backend process them. Additionally, output from the backend is stored as JSON in S3 so the frontend can retrieve for display purposes.

Challenges we ran into

The main challenge we faced was learning how to make our frontend and backend communicate. With the help of mentors, from Telstra, Atlassian and Australia Post they provided us insights into solving our main issue. Though we did not quite get everything integrate into a single working piece of software.

Learning aspects of AWS was also challenging considering no one on our team had any prior experience.

On top of that applying TensorFlow and Keras in a full project context was challenging in terms of the lack of resources (hardware) and training data was a timely process.

Accomplishments that we're proud of

Despite not completing a functioning prototype at this point in time, we are proud that we delved into new software, tools and packages that we never had prior experience with and tried our best to utilise them. Finally, we are proud of how we conducted ourselves as a team, given the diverse nature and range and variation of skills and knowledge.

What we learned

First of all, the importance of communicating as a team is crucial. Main points include team ideation, being critical and delegating appropriately according to each team members strengths. Another point is learning to approach mentors or team members when you are struggling. Overcoming the stigma or anxiety of admitting being ‘lost’ is important lesson, and we found when we overcame these barriers, we were able to progress.

What's next for Sense+

At the moment the Sense+ remains at its core an idea, not necessarily a piece of deliverable software. In the future we seek to improve upon accuracy when analysing and detecting emotion. This includes but isn’t limited to; more sophisticated sentiment analysis, improving the modelling and taking advantage of other bio-metrics that may come with the advanced of technology such as detecting heartbeat etc.

In terms of reach and usage, possibly uses is that companies could employ such software to monitor the well-being of employees. In the future the software could be more passive so that individuals can be monitored (of course with consent and confidential) in a more natural manner. This would yield accurate information on employee well-being rather than self-reports where people may lie because of stigma and fear. This could greatly boost the overall productivity and mental well-being within the company.

Other sectors this could be applied in is hospitals and education.

+ 69 more
Share this project:

Updates