As hundreds of thousands of people continue to lose family members due to the pandemic, our team has reflected on how we may one day say goodbye to those we love. In addition to seeing families grieving from their losses, we have seen many public figures pass away both before and during the pandemic, such as Kobe Bryant’s untimely death. We wanted to make use of AI and ML to help people cope with their losses, both with loved and inspirational ones.

Thus, we took a unique approach to the idea of staying connected during quarantine. Instead of creating an application that connects users conventionally, which already has numerous solutions, such as Instagram and Facebook, we opted to develop an app meant to connect people with those they cannot connect with anymore -- namely, those who have passed away. We wanted to find a way to allow friends, family, and even fans to artificially interact with these people and keep a memory of their experiences with those they have lost.

Thus, we developed Resurrect, an application that allows users to bring back those who have passed away through Generative Natural Language Processing Models, which replicate distinct personality and conversational traits.

What it does

Resurrect is a unique progressive web application that lets users artificially bring back loved ones and late celebrities. The core of our project is multiple generative NLP models, which allow users to build conversational agents with the personalities of anyone the user desires. Note that our demo shows the process, but because our model is incredibly large and because each bot needs to be trained for hours, the only way for us to deploy the generative models is to gain access to expensive, high-powered servers. Our model will work for anyone so long we are given the required data, and we hope to automate our model by gaining access to these servers.

First, users can upload both a csv file of their downloaded conversations (by connecting a phone to computer) with their late family members, as well as an audio clip of them speaking. The file is then uploaded to Google Cloud, and then we input it into our model. Our model takes the text conversation and begins training to replicate their speaking style. We found a research paper written by Google AI researchers on voice cloning, and while we were unable to implement this model with our generative models due to not having server power, we can still take the inputted audio sample and generate new samples. Once we finish processing their files, we notify them via email. We return a generative model embedded within a messaging interface, where they can have an interactive conversation with their late friend or family, allowing them to cope with their loss or to relive their memories.

We also developed a model to allow people to interact with late celebrities. Rather than using messaging transcripts, we web-scrape data from their twitter accounts, which often display their personality and resemble how they interact with fans. If a celebrity model has already been created, users can instantly interact with it, but if not, they can request one and we will quickly develop one for them.

Lastly, we have a page for those struggling with the death of their loved one or any other mental health issue. We created a generative model that acts as a counselor and therapist for mental health patients. This creates a supportive environment for those struggling with tough losses. We also have a forum where people can express themselves and meet new people who are facing the same grief. In addition, we created an interactive map using the Google Maps API, which allows users to find local therapists to receive additional support from.

How we built it

After numerous hours of wireframing, conceptualizing key features, and outlining tasks, we divided the challenge amongst ourselves by assigning Ishaan to developing the UI/UX, Adithya to connecting the Firebase backend and creating the chat features, Ayaan to developing our base generative models and researching the voice replicating model, and Viraaj to training the models with AI-Fabric from UI Path and connecting the backend.

We coded the entire app in 5 languages: HTML, CSS, Javascript, DockerFile/Makefile, and Python(Python3 /iPython). We developed our chat interfaces and integrated our models using PythonAnywhere and the Flask Framework. We used PythonAnywhere as our backend. We used Javascript to create our website backend, and used Google Cloud to store our data. We hosted our website through Netlify and Github.

For this project, we focused on developing generative NLP models with Pytorch and Tensorflow. For all our models, we used the pre-trained HuggingAI generative model and fine-tuned it on our data for each circumstance through transfer learning. For our counselor bot, we used the pre-trained Bert NLP Model and fit it to our data. When we get a message from the user, we are able to convert it into a latent vector and thus generate the correct output message. For our voice cloning model, we followed the documentation in the Google AI Research paper, and we were able to recreate their results with modifications, but couldn’t integrate due to server restrictions.

In order to collect data for these models, we developed two webscrapers. First, we created a basic web scraper to collect and format tweets based on a twitter handle. We then developed a web scraper to web-scrape, a forum for experienced and qualified counselors to answer questions, provide support, and post advice. For our adaptation model, we downloaded CSVs of text conversations from our iPhones and used them as data.

Challenges we ran into

The primary challenge that we ran into was developing our generative models. Since we have never built any generative NLP models, we weren’t sure how to start. Luckily, we found great documentation on how to develop them. We ultimately built 4 generative algorithms that all have different tasks. Training these models was also a huge challenge, and we saw that it was taking a long time to train. While we were not able to deploy our models, as they are too large to deploy on free and available servers, as long as users give us the CSVs or twitter handles, we can develop a bot for them.

Accomplishments we are proud of

We are incredibly proud of how our team found a distinctive yet viable solution to allowing people to cope with the loss of their friends and family. We are proud that we were able to develop some of our most advanced models so far. We are extremely proud of developing a solution that has never been previously considered or implemented in this setting.

What we learned

Our team found it incredibly fulfilling to use our Machine Learning knowledge in a way that could effectively assist people who have lost their friends and family. We are glad that we were able to develop a wide range of generative models to help a vast range of people. Seeing how we could use our software engineering skills to impact people’s daily lives was the highlight of our weekend.

From a software perspective, developing generative models was our main focus this weekend. We learned how to effectively build generative NLP models and web scrapers. We learned how to use great frameworks for ML such as Docker/Makefile and Flask. We grew our web development skills and polished our database skills.

What is next for Resurrect

Since our application is free and available to the web, our project can be scaled and implemented anywhere and with many other programs. With the possibility of a second wave for COVID, it is imperative that people have access to resources that can improve and stabilize their mental health and help them cope with the losses of their loved ones that are inevitable even beyond COVID.

In terms of our application, we would love to deploy our models on the web for automatic integration. Given that our current situation prevents us from buying a web server capable of running the models, we look forward to acquiring a web server that can process high-level computation, which would automate our services. We would also like to find new ways to collect datasets for our adaptation model. Lastly, we would like to focus on refining our voice cloning software and be able to integrate it with the rest of our models.

Our Name

We chose the name because our application attempts to resurrect a lost person and fill the space they left behind.

+ 1 more
Share this project: