Aida

AI Powered Virtual Assistant For the Elderly

Comment 1

Inspiration

According to the World Health Organisation, by 2050, the number of people aged 60 years and older will double to 2.1 billion. This statistic is not just a number; it represents the reality of our beloved grandparents and other elderly relatives who are struggling with daily tasks, emotions, and declining abilities.

We felt sad seeing our elders suffer, especially since they have been neglected in the tech race. With each new technological advancement, we have been focused on meeting the needs of millennials, while our seniors are left feeling alone and forgotten.

But it's time to change that. It's time to take care of those who were left behind. As a team, we have been inspired to use our skills and expertise to design a system that can assist the older generation. We don't want them to feel isolated and lost in this fast-paced world. Instead, we want to help them in any way we can as the younger generation. Our AI-powered virtual assistant, AIDA, is designed to provide seniors with the support and assistance they need to stay independent and maintain a higher quality of life. We believe that this will not only improve the lives of our elders but also bridge the gap between generations and foster a sense of connection and care.

WHY?

Background : Older people may experience changes in their brain structure and function, such as reduced blood flow and decreased synaptic density, which can lead to memory decline. Additionally, older individuals may have an increased risk of developing age-related conditions, such as dementia or Alzheimer's disease, which can further impair memory function.

Solution: Aida has a ChatBOT which effectively works as a memory aid by providing personalized responses based on previous messages and context. We also implemented a Recall a Memory feature which allows the elderly to quickly get the information they may have difficulty remembering. A Google but just for you!

Background: Older people may experience age-related changes in their vision, such as presbyopia (reduced ability to focus on close objects) and reduced contrast sensitivity. They may also have age-related eye diseases, such as cataracts or macular degeneration. These factors can make reading documents more difficult and require the use of reading aids or larger font sizes

Solution : We have build a OCR pipeline, where the user can upload any type of document like handwritten-notes, Tax documents etc and then ask AIDA questions about the document to understand it in a conversation way or use our search box to query data.

Background : Conversation provides an opportunity for social interaction and connection, which is particularly important for older adults who may be experiencing social isolation or loneliness. As individuals age, there can be changes in the areas of the brain responsible for processing emotions, such as the prefrontal cortex and the amygdala. These changes can affect the ability to recognize and understand emotions accurately and lose context or retain information from a meeting:

Solution : Whenever you place a call through our app, Aida transcribes the meeting and gives you a summary about the meeting, not just that, you can go to the chat app and ask it questions about the meeting days later and it will know. We have also implemented emotion detection, that gives you an idea about how the other user's feel that might help you engage more in the conversation.

Background : Language barrier means millions of elderly can’t access all the technological progress we are collectively making as a society:

Solution: Our app is completely language agnostic, meaning whenever you have input data, be it scanning document or talking to Aida, you can talk in 99 and Aida will have your back.

What it does objectively:

OCR: The OCR Scanner can be used to scan any document or hand-written notes and is stored in our elastic search server. Users can ask queries about this document or instruct Aida to summarize it. For example, if the document uploaded is a bill, the user can ask the assistant when the due date is.

VideoChat: Deficits in recognizing emotion is a major issue in amnesia and dementia patients. Our video chat implementation assists the user to determine the emotion of the caller. Moreover, a transcript of the meeting is stored and the user can ask questions about the meeting later. (remove emotion checker?)

Voice Note: The voice note feature is available from every screen to log in any information the user would want stored with multiple languages supported. The voice note feature is accessible from any screen on the app to log in any information to haystack by speaking into the mic. This feature can be used to perform various tasks like setting reminders or changing dosage of medicines.

Recall a Memory: The recall feature is available on every screen and enables the user to ask questions about any of the information they have logged in through the OCR, voice note, medicine logger, and video chats.

ChatBOT: The chatbot allows users to have a conversation with the AI-powered virtual assistant and can provide support, answer general questions and questions about the information they have logged in through other features.

Medicine Logger: Users can upload their weekly medicine schedule with medicine name, dosage, and time. Their daily medicines are shown on the view medicines screen. (mind blank about food and ui)

How we built it

AWS diagram

The app was implemented with specific design choices to cater to the elderly. A calm experience for the users was one of our priorities. So, we implemented a neat, simple and intuitive UI without any clutter so the app is easier to navigate for them. Furthermore, we chose a pastel colour pallet that represents calmness and neutrality.

Front-end: The Front-end was designed keeping an optimal user experience in mind. We utilized PWA technology so our app is accessible across multiple devices. Responsive Design was prioritized as well to ensure our app looks great. To create the user interface, we used ReactJS.

Back-end: We have utilized Haystack, OpenSearch and Firebase real-time database in the Back-end. Whenever a user sends a query, the OpenSearch is used to search for any previous context to provide personalized responses to our users. The Firebase real-time database stores the medical logs of the user and used their authentication feature for login.

Aida-API: The API was built using Fast-API and deployed on AWS Lambda and served with API gateway

The voice note feature, OCR, and Medicine Logger save the user's entries to the OpenSearch. Queries are directed to AIDA-API, running on an Amazon EC2 instance. Personalized responses are generated using OpenSearch to locate any relevant information and GPT-3.5 to formulate a response. Amazon S3 stores images for OCR, which are transmitted to AWS Lambda for data extraction via Amazon Textract. Extracted data is then saved to the OpenSearch DB.

We built the VideoChat feature using a Web RTC serverless implementation and firebase to establish a signalling server. Our emotion detection algorithm operates by identifying the primary subject in the frame with the highest probability, and detecting their emotions through a CNN with 67% accuracy. To record the meeting's dialogue, we utilize Open AI's Whisper to generate a transcript, which is then stored in OpenSearch. Additionally, our VoiceNote feature employs OpenAI's Whisper API to convert SpeechToText and save the resulting document to the OpenSearch.

Using OpenAI's Whisper API, our VoiceNote feature transforms spoken words into text, which is then stored in the OpenSearch DB. The Medicine Logger tracks the user's medication intake and records it in both the Firebase real-time database and the OpenSearch DB. Our Recall a Memory feature and ChatBOT rely on the OpenSearch DB to retrieve any relevant information based on the user's query and prior context, with GPT-3.5 generating a response accordingly.

Challenges we ran into

During the development of the Video Chat feature, one of the significant hurdles we encountered was the setup of the WebRTC infrastructure and signaling server. We had 5 developers for 24 hours, = 120 hours of development time to implement a fully scalable app on the cloud. It was a challenge, delegating tasking and understanding who can manage which feature, but we managed to almost hit all our goals.

Accomplishments that we're proud of

Successfully being able to understand the issues of the target audience who are usually overlooked and come up with great solutions for it was a great learning opportunity.

Deployment of all servers , APIs, endpoints in the cloud using AWS, firebase and haystack. This deployment structure allows our app to scale effortlessly and handle increasing user demand.

Implemented the retriever-reader architecture through Goggles Bert Language model and using AWS elastic search as the retriever to get the context and Open AIs gpt3 model as a reader effectively.

Being able to successfully implement the WebRTC protocol for video chatting.

What we learned

Learned extensively about elastic search, API development, UI/UX design, cloud architecture, AI model development, and WebRTC.

What's next for Aida

As an AI-powered assistant specifically for the elderly, Aida has the potential to impact the quality of life of many people. The back-end architecture is specifically designed to be highly scalable and cloud-based making it well-suited for expansion. The future of Aida also relies on ongoing development and improvement. We want to continue gathering feedback from users to refine and enhance the features and functionality of our app.

Share this project:

Updates