SilverServant

The Silver Lining of This Digital Age

Comment 1

Inspiration

We felt that more could be done to cater for the rapidly ageing population of our nation, especially those of the pioneer generation. Based on some of our own personal experiences, we have come to know that there are many elderly in Singapore who live alone, due to reasons such as their children leaving them to live on their own, or if their spouse/partner has passed on and many similar reasons. These elderly often live in a 1 room flat and have little to no social interaction with the people around them, even neighbours. As such, the mental health of these elderly individuals often are not in the healthiest state, and mental illnesses such as depression tend to arise commonly. Furthermore, with no one around to take care of them, what would happen if they were involved in an accident at home like a fall? For social workers taking care of elderly individuals, it can also be quite difficult to keep tabs on them and if an accident happens, the worker may not be alerted fast enough. SilverServant was thus created to aim to serve these needs, to address these pain points using one integrated solution that is end-to-end: from elderly individuals to social workers.

What it does

The first point of interaction comes when the elderly individual interacts with the social worker. The elderly’s profile and demographics are taken down and stored in our database. From there, the elderly individual is then given access to Silver Friend, our Chatbot Companion. The profile and demographics are fed to Silver Friend, who will use it as context to refine the way it communicates with and interacts with the elderly individual. Based on the profile of the individual and their mental illnesses, Silver Friend can schedule timely check-ins (such as every 6 hours) to check-up on the individual. As the elderly individual interacts with Silver Friend, the individual's responses will be recorded and monitored to see if there are any warning signs in their speech that could lead to potential self-harm. This is done through using our Natural Language Processing Models, where the responses are run through our model and a sentiment is produced. Currently, the 2 sentiments it produces are for suicidal and non-suicidal. In the event of a scenario whereby the responses of the elderly individual are deemed to be suicidal, the social worker will be alerted and further action will be taken from there.

How we built it

** Silver Friend (Chatbot Companion): Our Chatbot Companion makes use of LLMs such as GPT-3.5 to formulate responses to the user’s text input. Based on the description of the elderly individual, it is fed into the LLM as context for the Chatbot to understand more about the individual it is talking to, and allow it to refine its responses to cater for the elderly individual at hand. Our Chatbot uses WebKit Speech Recognition and Web Synthesisers to provide speech-to-text and text-to-speech capabilities respectively. Furthermore, our Chatbot uses JavaScript’s native setInterval() to schedule when messages are to be sent automatically to the elderly individual to check up on them. With GPT-3.5’s intrinsic ability to cater to multiple languages as well, we made use of this to make Silver Friend able to interact with elderly individuals who don’t speak english, but speak either mandarin, malay or tamil.

** Natural Language Processing (NLP) Models: Our models used can be split into two categories: pre-trained and self-trained. The pre-trained models that we used in this project are NLTK VADER and Huggingface BERT. These models were used mainly for sentiment analysis of text inputs generated by the user in the chatbot environment. Additionally, we also trained our own model using the Long Short-Term Memory (LSTM) Recurrent Neural Network (RNN) architecture to predict more specific categories of speech like depression and suicidal tendencies. These models were trained from labelled datasets sourced online on Kaggle. The models were both trained with about 20,000 labelled data each. With these models built, we split our processes into three categories namely: Sentiment Analysis, Depression Analysis, and Suicide Analysis. For Sentiment Analysis we used an ensemble of the pre-trained models to give us a prediction of basic negative/positive/neutral sentiments. For the Depression Analysis, we used an ensemble of one pre-trained model (BERT) in combination with our own model built to predict depression. And lastly, for Suicide Analysis, we used only the model built to predict suicidal tendencies. Our decision to use LSTM instead of other architectures is due to LSTM’s ability for sequential data handling which are well-suited for capturing contextual information and relationships between words in a sentence. We also liked its long-range dependencies which can similarly be important for understanding context in natural language. They can also maintain information over longer time intervals, which can be useful for tasks that involve understanding context in longer paragraphs.

** Fall/Inactivity Detection: It’s built using python script that utilises computer vision and the pre-trained YOLOv3, real-time object detection algorithm that identifies specific objects. Humans are specified to be the target detection, and our own unique methodology of detecting fall is by analysing the size difference between 2 human frames. If the size of the human frame decreases suddenly by more than 65%, a fall is recognised and a screenshot is taken and sent to the dashboard, for the admin user/social worker to verify if a fall has indeed happened. While not the perfect methodology, it provides an accurate detection of possible falls.

Challenges we ran into

** Silver Friend: Finding the right tools to allow Speech-To-Text was quite challenging, and integrating it with the Chatbot was not easy initially. Eventually, we managed to break down the entire task of speech-to-text integration into smaller functions that were done separately so that it became easier to integrate.

** NLP Models:

We found that training models properly takes a very very long time (more than we expected). We also learned that it is not enough to have massive amounts of data, it is also very important to fiddle and tweak with the hyperparameters and to test extensively in order to get a good prediction. These were the main challenges we ran into when building the models, they just weren't accurate and reliable enough. We also faced the challenge of putting models together as an ensemble model as it was our first time doing such a task. Another unexpected challenge was ensuring that our data was properly formatted as some data were incorrectly formatted, resulting in errors in our code.

** Fall/Inactivity Detection: It is a complex problem to accurately detect falls, which normally requires more confirmation signals to reduce false positives. However, this provides a good starting point, with a decent enough accuracy and low false positive rate to detect potential falls. We can definitely explore more robust methods of doing such detection, especially using them in tandem with other supporting measures/devices.

Accomplishments that we're proud of

One of our standout achievements is the successful implementation of Silver Friend's ability to communicate in multiple languages. By enabling interactions in English, Mandarin, Malay, and Tamil, we've created a solution that is truly inclusive, accommodating the diverse linguistic backgrounds of Singapore's elderly population.

We also took pride in creating a personalized experience for each elderly user. Our Chatbot Companion, Silver Friend, utilizes the individual's profile and demographics to tailor responses, resulting in more relevant and empathetic interactions. This achievement reflects our commitment to providing meaningful connections for those who may be isolated and to create a technology-driven solution that doesn't sacrifice the human touch. By combining AI-driven interactions with features like check-ins and early warning sign detection, we've humanized technology for the betterment of mental health support.

For the NLP models, we are proud that we managed to train two of our own predictive models from scratch in one day and to get them working to a reasonable degree, despite our many limitations. We are also proud that we managed to combine models in an ensemble to better suit the context of situations we are trying to serve and predict. We thought that did a reasonably good job, considering none of us had prior experience in creating ensemble models, which we learnt reaps better results for prediction accuracy.

What we learned

Through the development of SilverServant, our team gained valuable insights and lessons that have deepened our understanding of both innovation and the challenges faced by elderly individuals in our society. We recognized the profound impact of social isolation on the mental well-being of elderly individuals. Many of the elderly population, especially those living alone, suffer from limited social interactions, leading to mental health issues such as depression. This highlighted the critical need for technology solutions that foster connection and companionship.

Designing Silver Friend to communicate in multiple languages was a significant step towards inclusivity. This feature allowed us to cater to Singapore's diverse ethnic population and highlighted the significance of considering cultural and linguistic diversity when developing solutions for a broader user base. Next, working with Natural Language Processing (NLP) models, including both pre-trained and self-trained, also emphasised to us the potential of technology in analyzing sentiments and recognizing signs of mental health issues.

In terms of ethical and privacy considerations, we were also reminded of the ethical and privacy implications of creating technology for vulnerable populations. Ensuring data security, informed consent, and the appropriate handling of sensitive information were key aspects of responsible development, and this was something that we gained more insight for as we embarked on this project.

As for NLP, we learnt that we should train multiple models with different architectures in order to compare their prediction accuracies. This is very important as different contexts and prediction goals would entail different model requirements. We also learnt that we should plan ahead in terms of effectively adjusting our models’ hyperparameters by exploring methods like Grid Search and Random Search which are systematic methods of finding possible hyperparameter values for each hyperparameter in a model. We also learnt that model accuracy is not everything as well, as we also have to consider model sensitivity and how to balance everything to create an effective model.

Overall, SilverServant taught us that technology has the power to bridge gaps in mental health support and companionship for the elderly. It demonstrated the potential for AI to provide tailored assistance and identify early warning signs, while emphasizing the necessity of responsible, ethical, and user-centered design.

What's next for SilverServant

Hardware-wise, we envision for SilverServant to come to life in the form of a physical companion robot, to provide a more tangible and personal sense of friendship and companionship to the elderly individual. This physical companion robot will be roughly the size of a small toddler. This robot will contain all the features that we have so far, and we believe the feature that will benefit the most from this would be the chat companion, as this would make the interaction with the elderly more human-like, which is pivotal for mental health. Software-wise, with regards to the chatbot, we plan to continuously improve its human-like qualities and responses to ensure that it provides the most human-like experience possible with the elderly individual. Next, we understand that mental health is never definitive and can vary from person to person, so we plan for SilverServant to be trained with more data on different types of mental illnesses such that it is able to cater to a wider variety of elderly individuals with different mental illnesses. As well, we plan for it to be able to identify more physical injuries or accidents, such as if the companion bot notices that the elderly individual has not moved for over 12 hours and to report it to the social worker or the authorities automatically. For our NLP models, we plan to train our models with real user inputs in order to better capture the context and patterns of elderly speech in Singapore in order to create more robust predictive models. We could also explore engaging local psychologists or the Ministry of Health to learn about the unique facets and aspects of mental health in Singapore and what troubles the elderly, specifically in Singapore. These new data could potentially make our project a potent form of digital healthcare for the elderly.

Built With

Share this project:

Updates