VRPersonalBanker

Virtual Reality Banking Assistant Initial Screen
Virtual Reality Banking Assistant Language Screen
Virtual Reality Banking Assistant Transaction History Display Screen

Inspiration

I personally hate making a trip to bank due to heavy traffic and waiting in line to get things done. Also having worked in IT division of Banking Industry in past, wanted to improve the pain points which Banks are facing, from my experience

Need to increase bank staff to serve customers to reduce customer waiting time and improve customer interactions at Branch. Needing experienced bank staff to drive discussions based on customers mood at the time of interaction
Providing multi-language support and needing physical branch presence in the city to provide the branch services
Capturing customer feedback to improve banking services at branch
Bankers spend lot of time in doing A-Z Reviews, helping customers to complete their KYC and making suggestions to the customers to choose personal financial goals, while their experience can be used to provide specialized services.
As per Banks Usability and Accessibility regulatory requirements, banks should support differently abled customers. It’s often challenging for abled person to make a trip to the bank.
Build once and run anywhere i.e. application can be accessed via URL from web browser on any device without any special hard ware / software requirements
No need to wait for scheduling the changes without impacting customers, with republish the new updates are automatically available using the same URL.
Pay per use cost
Helps Banks to be up to date with the market competition by introducing this new channel of banking and achieve customers satisfaction
Right now there is no such app in the market!
Horizontally scalable i.e. can be used for several banks!

What it does

VR Personal Banker provides Virtual Banking Assistance to Bank customers to complete A to Z Review, Block their lost or stolen card, place cheque book request, stop cheque, see their transaction history, pay bills, make funds transfer between their own account, get goals recommendation, at the convince of their home / anywhere using any device without making a trip to Bank or waiting in line.
Improved customer satisfaction and customer interactions by creating unique and new customer experience and conversation driven based on customer mood / persona
Apart from verifying the user info this application provides additional security by comparing the user image against the image saved in user profile, if the image doesn’t match after user verifies their details, it won’t let the user continue.
It provides multi-language capability, especially when the user is traveling and he needs local language preference in different country.
Recommendation engine provides goals suggestion based on user age and income
Feedback collection, which is lacking at Branch will help banks improve their services
Audit trail of each user activity helps in achieving regulatory compliance requirements
Capturing user emotion for each activity will help banks analyze the data further using services like AWS Comprehend for sentiment analysis and improve banks services
Respect Customer confidentiality i.e. customers don’t have to share their personal information with Banker when they are not comfortable answering few personal questions while completing their A to Z Review
Host coordinated movements with customer head movement and host eye movements provides life like experience.
It will be useful for vision impaired customers who can do all their banking by talking to the virtual banker without having to physically visit a branch.

How I built it

Interface with Lex run time without needing state machine
Convert the speech to selected language preference and play the speech to customer
Web RTC API - For capturing camera feed
Host head movement according to customer head movements
Amazon Rekognition - Rekognition Detect Faces to identify its Human interacting with the application. Analyze Face to identify customer emotions. Rekognition Compare Faces to compare customer image against what is stored in customer profile to confirm the user identity. This provides additional layer of security.
Amazon Translate - Translate the Lex response to user selected language preference. Setting the host voice id based on language selection
AWS Lambda -Lambda to process the request based on user input, store the data and return the response
Amazon DynamoDB - Storing customer information, transaction details, customer to account mapping, customer to card mapping, customer to payee mapping, feedback, audit trail, stop Cheque details, order Cheque book details
Amazon S3 - Storing customer's image for image comparison after user confirms his details
Amazon Lex - Accept user input and invoke the intent
Amazon Polly - Lex text response to speech conversion
Amazon SES - Sending notifications to the customers when a new account is opened, A- Z review is complete, bill payment is done, transaction is complete, Cheque book request is placed, card is blocked and when the stop Cheque request is placed
AWS Congnito - Identity Pool Id for authentication

Challenges I ran into

Synthesize Lex response to support multi-languages before playing the speech to the customer. No control over synthesizing the response when State machine is used, so I used System Bus and emit and listen to events to process the response.
Translate supports latest SDK whereas Sumerian uses 2.17 – Changed SDK URL to latest, after I learned Translate needs latest SDK
Sending Camera feed to Sumerian – implemented Web RTI to achieve this
Implementing Image Rekognition and Comparison – Promises to ensure the response is returned before continuing with the flow
Storing user emotions without using Lex Post / Post Content
Storing and Sharing session variables across multiple Intents
Hitting image Rekognition comparison API limits
Lex to accept multi languages as input – TranscribeService initialization is not working as it says it’s not constructor.

Accomplishments that I'm proud of

Sending Camera Feed to Sumerian for processing, customer head movement and identifying user emotions
Implementing Face Rekognition to confirm Customer identify
Implementing AWS Translate to translate Lex output up to any one of 15 languages selected by the customer
End to end integration with Sumerian, Lex, Lambda, Dynamodb, S3 and SES
Showing data in HTML content dynamically

What I learned

Capturing camera feed through Sumerian
Making calls to Face Rekognition and Face Comparison from Sumerian
How and where to use Rekognition Comparison to avoid throttling
Implementing Lex in Sumerian without using state machine to control the responses from Lex
Using AWS Translate to translate Lex response
Sharing session variables across intents
How to effectively create Hash Keys for Dynamo DB for querying

What's next for VRPersonalBanker

Implement transcribe to implement voice to text conversion for accepting input in multi-language
Implement AWS Comprehend to analyze customer sentiments and generate a report so Banks can improve their services offered.
Add more functionality to cover all the activities done at Branch through this application
Incorporate text based user input for the hearing and speech abled customers