Flood Victim Evacuation VR

Inspiration

Floods cause more than $40 billion in damage worldwide annually. In the U.S. losses average close to $8 billion a year. More than 1,200 people have died across India, Bangladesh & Nepal as a result of flooding, with 40 million affected by the devastation. At least six people, including two toddlers, were among the victims in and around India’s financial capital.

We honestly do not have enough Flood Responders. India is already a huge country with a massive population. Many people get affected by floods. We can create our own heroes by training them in VR.

What it does

This application is ideally best to experience with a VR headset. It is a virtual environment which has gone through a massive flood. There is a house near you that you have to walk into to find an affected victim, a little girl.

Arrows on the ground inside the house help you navigate and reach the little girl. Once you go near the girl, you have to talk to her and make her stress level reach a certain level for her to trust you and get evacuated.

The point is not to have a conversation but what to say in the conversation to decrease her stress and make her trust you.

How I built it

This project had to phases into it.

The first was the client side VR application to be built using Unity 2019.1 It was tough to get a flooded area so some assets were purchased, the rest were made using Unity plugins for particles and fluid mechanics. One the VR world was done, then came the speech integrations.

https://github.com/pourabkarchaudhuri/flood-sim-azure-ai-hackathon-2019

Here Azure Speech was used to its fullest using its Text-To-Speech, Speech-To-Text, Custom Voice, Custom Voice Endpoint Hosting and so on.

The next part was to create a web service for a Python based backend using Flask for the Unity App to talk to my server. On the server there should a module that handles conversations, stress, and reactions that the avatar in VR is supposed to give. Here the LUIS connector is written that does Intent Classification, NLU and Sentiment Analysis and sends it back to the Unity App.

https://github.com/pourabkarchaudhuri/empathy-engine-luis

Challenges I ran into

There were multiple challenges that had to be overcome.

The world had to feel realistic in VR, the avatar had to look dirty and scared, the conversation had to be unstructured. It was a very tough mix combining Cloud, VR and AI. But upon completion this turned out extremely well.

The next challenge was responsiveness. I had to put the Azure Service Instances nearer, optimize code and remove any blocking threads so that the conversation could happen as smooth as possible.

Accomplishments that I'm proud of

This simulation has a few features which I'm very proud to have since they completely add more flavor and value to the essence of the training and the VR experience as a whole.

The Voice of the Avatar is of one of my friends. This was done using Azure Custom Speech where I trained her voice samples. This adds more value and more impact if the first responder has to rescue a known person, they would have to be very serious in doing so!
The Lips of the Avatar move to the real-time text to speech services using Azure Speech APIs to add more realism to the conversation
I added an element called stress which is based on Sentiment Analysis. Prolonged conversations piss her off, repeating the same thing makes her stress go up and she shouts back. It adds a different element of conversation.
The last was the dynamic water. :)

The point was not to have a conversation but how to have the conversation. These features really bring that out.