A study by UC Davis shows that music has the ability to activate regions of the brain responsible for autobiographical memories and emotions. We experience this phenomenon when a song makes us feel nostalgic, or reminds us of a person or event. We wanted to use this to help those with Alzheimer's remember things through music. Some members of our group have several members of our families, such as Prasann's grandpa, who are victims of neurodegenerative disease. We know more than many how painful it is to have your loved ones lose their memory, forgetting what makes you unique in their eyes, and we strongly believe that any efforts taken to alleviate the pain are well worth the effort.
What it does
When the user is at a place they would like to remember, they can place a marker on the map. This opens up a prompt for them to write a short description of the place, a memory they had there, or something they saw. Our app then uses an AI sentiment classifier to determine the user's feelings towards the location and automatically finds and plays a song aligned with their mood. When the user goes to the location again, the app will send a text notification reminding them to check the app and listen to the song again, look at their pic, or read what they wrote for that location. Users can also explore all the markers they've placed on the map, exploring what they took note of at each of those places. After a few visits, the song may become linked with the memory, and just hearing the song should remind users of the location.
How we built it
- Frontend built with Flutter
- Flutter allows for beautiful UI along with quick testing and deployment
- Backend built with Firebase, Twilio, Flask Server, and Google Cloud NLP
- Firebase was stored User Data
- things like images, urls, and text data on it
- Twilio was for notifications on geo-location to try to notify people when they entered an area where they submitted a note.
- Flask hosted the sentiment classifier and the music website webscraper.
- Google geo-location to track when / where users encountered locations where they had experienced past memories.
- Google Cloud NLP Sentiment classifier allowed us to provide music to go along with memories, making them both more memorable and more enjoyable!
Challenges we ran into
- We ran into issues with webscraping for songs because of how slow it was initially. We optimized this process by creating a rudimentary database and using dynamic updating which sped the time required to mere seconds.
- We also ran into issues with Google Cloud's Natural Language API since there was no direct functionality with Flutter. To work around this, we had use a Flask server to interface as an API, and then used that to communicate with our app.
- We had issues with integrating the Twilio API into our app due to confusion about body parameters and authentication - thanks mentors!
Accomplishments that we're proud of
- UI is flames. It's leaps and bounds better than some of the other apps we've created in the past.
- We were able to interface a bunch of APIs.
- Some of them didn't have direct support for Flutter so we had to use some workarounds
- Others we spent some time trying to figure out how to get to work together properly, such as Twillio and Google Maps.
What we learned
We learned how to use Cloud NLP with GCP, the GCP SDK, webscraping using Selenium, and Twilio. We also learned how to use a lot more Flutter widgets which helped make our UI come to life.
What's next for Memory Notes
In the future, we would love to add the ability to share your memories with friends, as well as have to option to create memories about friends and loved ones. It would be interesting for people to see what songs their friends associate them with.
Who we are:
Lucas Cai - LucasVegetable#1114
Prasann Singhal - PrasannSinghal#9723
Samuel Yuan - SamuelYuan#3585