1502-SF27 Alive

talk to an avatar of a person who passed away
come back to your room from your childhood

Inspiration

Our team initially worked on developing avatars for a large platform, where they were used for commercial purposes. After the project launched, a private client approached us with a unique request: to create an avatar fully trained on his voice, information, and memories, which would communicate with his sister after his death. This led us to create a separate app that allowed the client to train the avatar, enabling real-time communication with his sister. During this process, we realized that what was missing was immersion—an experience that allowed people to truly reconnect with memories and engage with those they could not communicate with in real life. This sparked the idea for a psychological assistant app designed to help users find a comfortable space for self-reflection and dialogue with avatars of people they couldn’t speak with in person. In addition, multiple studies have shown that familiar environment and experiences have positive therapeutic effects to over 55 million patients of dementia such as Alzheimer’s disease, and we believe our app will be an easy-to-use care method for the patients, their families, and their medical team. Virtual reality, with its ability to offer a fully immersive experience, seemed like the perfect platform to bring these visions to life.

What it does

We create a virtual room based on the user’s childhood memories. Right now, it’s a prototype, but in the future, we plan to build typical rooms for each decade, filled with interactive items that people can recognize. The next step is to allow users to create their own rooms using 360-degree photos or software that lets them design their own unique spaces. In the room, there is a screen on the wall. When the screen is turned on, an avatar of the person the user wants to talk to appears. The avatar is created using data, photos, and videos that the user uploads before starting the experience. The more data the user provides, the more detailed and meaningful the conversation will be. The avatar can respond to questions in real time, using the same words and phrases the user is familiar with in everyday life. For the presentation, we are using a sample avatar named Alex, which is trained using tourism data. We started by developing the core technology behind the avatars using AI and machine learning. Our team used motion capture and facial recognition to ensure that the avatars could mimic human expressions and movements in a natural way. For the virtual room, we designed the environment using a mix of 3D modeling tools and VR platforms like Unity and Unreal Engine. This allowed us to create an immersive space that users could interact with. To build the avatars, we used a training process where users upload their data, including photos, videos, and voice recordings. This data helps the avatar learn to communicate in a way that feels personal and familiar to the user. We also integrated real-time response capabilities, so the avatars could answer questions instantly, using the specific language and expressions the user is accustomed to. As we moved forward, we continuously tested and refined our platform, gathering feedback from early users to improve performance and user experience. The prototype is now ready for demonstration, and we plan to continue developing more advanced features, such as customizable rooms and more dynamic avatars.

How we built it

For the front-end, we created a bot that can receive information and generate responses based on the data uploaded into it. We integrated this with D-ID Studio, where we had a pre-generated avatar, which we then transformed into a real agent capable of interacting with users. The virtual room itself was built using Unreal Engine, allowing us to create an immersive space for the user. This combination of technologies gave us the flexibility to create a dynamic, personalized experience where users can interact with their avatars in real time.

Challenges we ran into

One of the major challenges was that everything had to be created manually. We spent a lot of time searching for the right inspiration for the room and ensuring that the design reflected the user’s memories and experience. However, as we moved forward, we ran out of time to fully train Alex and refine the experience to match the final vision we had in mind. This meant that while the prototype was functional, there were still aspects of the experience we weren’t able to fully develop as we had planned.

Accomplishments that we're proud of

In general, the fact that we didn’t give up and kept pushing forward despite the challenges is a major accomplishment in itself. )))

We’re also proud of successfully integrating the avatar and placing it on the wall, which was a key feature of our project. Being able to make this work and bring it to life in the prototype was a significant achievement, demonstrating both our team's creativity and technical skill.

What we learned

That Unreal Engine doesn't support/digest heavy URLs

What's next for 1502-SF27 Alive

We are in the early stages of this project, and there are several key developments ahead. Avatar Cloning App The next step is to create an app where users can clone avatars of people they want to bring into the virtual environment. This app will allow users to upload videos, voice recordings (for voice cloning), and other personal data to train the avatars. The goal is to enable users to create avatars that closely resemble the people they wish to interact with, allowing for more personalized and realistic communication. Room Assets Based on Time and Location We also plan to develop a set of assets for rooms, designed to reflect different time periods and geographical locations. These assets will allow users to recreate environments that match specific memories or historical contexts. For instance, users will be able to design rooms inspired by particular decades or cultural settings. This will offer greater flexibility in customizing the experience to fit the user’s needs. These are the primary next steps, and as the project progresses, further features and improvements such as addition of the person’s own voice and music associated with them will be made.