MementoAI

Inspiration

This assistant has been initiated to function for patients with Alzheimer’s disease or Parkinson’s disease during stage 1 and stage 2 of their treatment. People with these diseases eventually develop a habit of forgetting their daily chores. We wanted to build an assistant that can help people during these times.

What it does

There are quite a few features that we have planned to incorporate into this model. The assistant serves to remind people of who their family members are through storing their images. The patient can ask the AI regarding who a particular person is and the AI will display the picture of the person using the images in its database. The assistant will be able to record user's conversations and store the summaries so that they might be useful at some point in the future. The AI serves as a reminder and also will be able to manage the patient's daily schedule through medications, daily chores, exercise and planning hospital visits. They will be taken as an input and will be notified to the patient through vocal notifications when it is time. Hospital visits are planned depending on the level of information the patient can remember.

How we built it

Python was the main language that we have used for this project. We've planned to make the assistant by integrating the different features of the product in individual environments.

Challenges we ran into

There were quite a few errors that we were facing when we were trying to classify the input text to the respective image of the person. It was mainly about the deprecation of the keras and tensorflow libraries. We've also faced a few issues while making reminders and vocalizing notifications. To be specific, we faced issues with Python's text to speech library "pyttsx3" as the engine to vocalize the notifications was not restarting on its own. We fixed it by adding engine.stop() after each reminder was done and we reinitialized it for the upcoming reminders. We were also stuck with the integration of Fetch.AI's agent into our text-to-speech recognition code.

Accomplishments that we're proud of

We're proud of how well we were able to combine many features, such as the ability to store and show pictures of family members, set and speak reminders, and summarize talks. Even though it was hard, especially with the text-to-speech engine and making TensorFlow and Keras work together, we were really close to making a prototype that works and can help people with Alzheimer's or Parkinson's disease.

What we learned

We felt really happy that we've learnt quite a lot during this hackathon. We learnt a lot about Large Language models and the way they actually summarize the entire voice calls that two persons have. We also came to know about building an interface that takes reminders as input and sends voice notifications to remind us regarding the same. We've got to experience a glimpse into Fetch.AI's contribution towards developing AI agents covering many fields.

In addition to learning technical skills, we also learned how to debug problems that come up when we use deprecated libraries like TensorFlow and Keras. We learned how to make tools that are easy for people to use and how important it is to make systems that meet the needs of people who have cognitive impairments. Another important thing we learned was how to use "pyttsx3" to handle voice notifications effectively. We also learned more about big language models and how they can summarize conversations. This helped us make our assistant more useful.

What's next for MementoAI

We will be making a complete prototype soon enough. Then we are planning to potentially make it a hardware system that can help us analyze the daily chores of people through a camera so that we get to track their attention span when they're working on a chore. If they tend to deviate, the AI assistant can remind them what they were doing.