SchizoFRIENDia

Inspiration

Avatar Therapy (AT) is an innovative experiential approach for patients to create an avatar of the voice in their auditory verbal hallucinations and allows them to gain control over their symptoms. Talking to these avatars also shifts control from the avatar to the patient.

Does Avatar Therapy work? Avatar therapy can reduce hallucinations significantly after 12 weeks (half of the normal treatment span) Avatar therapy has a more impactful effect on alleviating other symptoms of schizophrenia. In some cases, avatar therapy has even been able to completely stop seizures.

If it works so well, then why is it not administered worldwide? The treatment is administered through very experienced/specialized doctors (who played two roles- the mediator/supporter and the avatar ) — something that cannot be scaled because of the difficulty to provide specialized care for a high number of patients.

What it does

HUMANISES THE VOICE IN THE HEAD THROUGH A PERSONAL AVATAR:

Deep video synthesis (or deep fakes) can be used to augment the avatar process itself, and further the idea of bringing these persecutors to life.

CONVERSES WITH THE AVATAR IN THE PRESENCE OF AN AI THERAPIST:

Most of the main scalability and effectiveness issues can be resolved using Natural language processing, the category of AI that deals with understanding and processing the human language.

How we built it

DialogFlow Chatbot: The chatbot was built using DialogFlow. It involves a three-way conversation between the patient, therapist and the avatar. Fuzzy matching parameter extraction was used so that it can understand even if the user misspells a word or only enters part of the words from entry entities. Voice-to-text and text-to-voice activation is integrated as well, so that people who are disabled can also use this therapy.

Avatar Generation: The avatar generation using a deep learning model specifically a GAN model to give a face to the voice the person is hearing. The frontend communicates with a flask api. The patient sent an image which then is passed through the model and the avatar is generated. The avatar is sent to back to frontend.

Integrating it all in the web app:

We used React for the frontend and Flask for the backend which contains the avatar generation code. Image upload is handled by the frontend whereas the image transformation code is in the backend. The chatbot runs on the Kommunicate API which gets the data from DialogFlow.

Challenges I ran into

Avatar therapy that we have suggested is an ongoing research not implemented by anyone else.
We needed to come up with a deep learning model to make an avatar so that the patient can give a face to the voice they hear. By this, they are not afraid of them.
The therapist is an AI bot which we needed to train to make a conversation with the patient. The patient needs to fight back with the "voice", hence the AI bot tries to replicate the voice and helps the patient by re-creating the voices they have heard.

Accomplishments that I'm proud of

We are able to make the avatar using deep learning model
The chatbot is functioning according to the avatar therapy helping the patient.
Let’s be honest, the website is pretty !!!

What I learned

The deep learning model to create the avatar.
The chatbot feature to be able to have a conversation.

What's next for SchizoFRIENDia

Using capabilities such as sentiment analysis, and harnessing data from previous sessions, the entire process will be much more efficient and deployable by any non-specialized professional.