Mind Palace

mind palace of experiences

Inspiration

Our project aims to proactively combat mild cognitive impairment with reminiscence therapy. Up to 20% of adults older than 65 have the condition, according to the Alzheimer's Association. Reminiscence therapy helps people by affirming their long-term recall ability. People with dementia often feel frustrated with their short-term memory limitations. But, by focusing on things they can remember, reminiscence therapy can help people with Alzheimer's feel a sense of mastery over their memory and cognition skills. According to a meta-study, reminiscence therapy had a significant positive effect on depressive individuals in elderly people with dementia. Study We were inspired by the latest AI and VR technologies being showcased at TreeHacks, and endeavored to provide a use case for improving our understanding of our own life stories, as well as providing solace for elderly individuals who struggle to remember some of the most touching moments of their lives.

What it does

Mind Palace is a 3D AI chatbot in VR that enables you to record your own story, and live it back in an immersive way as if you were still there. The AI "Oracle" interviews the user with questions about a life experience they would like to share. The AI's follow-up questions are generated, providing a personalized authoring experience. When the user finishes telling their story, the Oracle presents the user with a generated summary of the key points of their story, as well as generated images depicting what happened. The Mind Palace becomes a collection of many life experiences the user has gone through. Mind Palace enables this with features like a library of prior memories, generation of 3D visuals that envelop the user, and voice-activated speech-to-text communication. We’re advancing the way not only stories are told, but how we understand ourselves through storytelling.

Features

Capture your memories with speech-to-text translation.
Converse with an AI character that interprets your memories.
Experience a summary of your memories with generated visuals and captions.
Benefit from engaging in reminiscence therapy with AI.

How we built it

We used Unity to develop VR interactions for Quest headsets, OpenAI's API for AI generating conversational text outputs and images, and Meta’s Voice SDK was used to translate realtime voice input into text––directly inside of VR.

Challenges we ran into

Implementing an AI algorithm within a VR experience was a particularly ambitious goal since we had to come up with a system for edging the two technologies into a shared experience.

Accomplishments that we're proud of

We're proud of having implemented AI into a virtual reality experience, and consolidating all of the above-listed features into a cohesive, intuitive user experience.

What we learned

We learned how to use the OpenAI API within Unity, and integrating it with voice recognition, text to speech, and realtime texture changes. Along with the technological changes, we gained an appreciation for the safe environment provided by VR to share our inner thoughts with ourselves.

What's next for Mind Palace AI

Mind Palace aspires to become the world’s leading solution for personal story generation that evolves the practice of reminiscence therapy to combat conditions like Alzheimer’s and dementia. This project was conceived to pioneer the emerging trend that its anticipated XR and AI will fuse together to enable immersive experiences that are intelligent, personalized, and engaging. We would like to expand to be used in real-world research, continue building out the product’s features, and contribute a positive impact to the larger community of people facing mild cognitive impairment.