Virtual Reality Banking Assistant Initial Screen
Virtual Reality Banking Assistant Language Screen
Virtual Reality Banking Assistant Transaction History Display Screen
I personally hate making a trip to bank due to heavy traffic and waiting in line to get things done. Also having worked in IT division of Banking Industry in past, wanted to improve the pain points which Banks are facing, from my experience
- Need to increase bank staff to serve customers to reduce customer waiting time and improve customer interactions at Branch. Needing experienced bank staff to drive discussions based on customers mood at the time of interaction
- Providing multi-language support and needing physical branch presence in the city to provide the branch services
- Capturing customer feedback to improve banking services at branch
- Bankers spend lot of time in doing A-Z Reviews, helping customers to complete their KYC and making suggestions to the customers to choose personal financial goals, while their experience can be used to provide specialized services.
- As per Banks Usability and Accessibility regulatory requirements, banks should support differently abled customers. It’s often challenging for abled person to make a trip to the bank.
- Build once and run anywhere i.e. application can be accessed via URL from web browser on any device without any special hard ware / software requirements
- No need to wait for scheduling the changes without impacting customers, with republish the new updates are automatically available using the same URL.
- Pay per use cost
- Helps Banks to be up to date with the market competition by introducing this new channel of banking and achieve customers satisfaction
- Right now there is no such app in the market!
- Horizontally scalable i.e. can be used for several banks!
What it does
- VR Personal Banker provides Virtual Banking Assistance to Bank customers to complete A to Z Review, Block their lost or stolen card, place cheque book request, stop cheque, see their transaction history, pay bills, make funds transfer between their own account, get goals recommendation, at the convince of their home / anywhere using any device without making a trip to Bank or waiting in line.
- Improved customer satisfaction and customer interactions by creating unique and new customer experience and conversation driven based on customer mood / persona
- Apart from verifying the user info this application provides additional security by comparing the user image against the image saved in user profile, if the image doesn’t match after user verifies their details, it won’t let the user continue.
- It provides multi-language capability, especially when the user is traveling and he needs local language preference in different country.
- Recommendation engine provides goals suggestion based on user age and income
- Feedback collection, which is lacking at Branch will help banks improve their services
- Audit trail of each user activity helps in achieving regulatory compliance requirements
- Capturing user emotion for each activity will help banks analyze the data further using services like AWS Comprehend for sentiment analysis and improve banks services
- Respect Customer confidentiality i.e. customers don’t have to share their personal information with Banker when they are not comfortable answering few personal questions while completing their A to Z Review
- Host coordinated movements with customer head movement and host eye movements provides life like experience.
- It will be useful for vision impaired customers who can do all their banking by talking to the virtual banker without having to physically visit a branch.
How I built it
- Interface with Lex run time without needing state machine
- Convert the speech to selected language preference and play the speech to customer
- Web RTC API - For capturing camera feed
- Host head movement according to customer head movements
- Amazon Rekognition - Rekognition Detect Faces to identify its Human interacting with the application. Analyze Face to identify customer emotions. Rekognition Compare Faces to compare customer image against what is stored in customer profile to confirm the user identity. This provides additional layer of security.
- Amazon Translate - Translate the Lex response to user selected language preference. Setting the host voice id based on language selection
- AWS Lambda -Lambda to process the request based on user input, store the data and return the response
- Amazon DynamoDB - Storing customer information, transaction details, customer to account mapping, customer to card mapping, customer to payee mapping, feedback, audit trail, stop Cheque details, order Cheque book details
- Amazon S3 - Storing customer's image for image comparison after user confirms his details
- Amazon Lex - Accept user input and invoke the intent
- Amazon Polly - Lex text response to speech conversion
- Amazon SES - Sending notifications to the customers when a new account is opened, A- Z review is complete, bill payment is done, transaction is complete, Cheque book request is placed, card is blocked and when the stop Cheque request is placed
- AWS Congnito - Identity Pool Id for authentication
Challenges I ran into
- Synthesize Lex response to support multi-languages before playing the speech to the customer. No control over synthesizing the response when State machine is used, so I used System Bus and emit and listen to events to process the response.
- Translate supports latest SDK whereas Sumerian uses 2.17 – Changed SDK URL to latest, after I learned Translate needs latest SDK
- Sending Camera feed to Sumerian – implemented Web RTI to achieve this
- Implementing Image Rekognition and Comparison – Promises to ensure the response is returned before continuing with the flow
- Storing user emotions without using Lex Post / Post Content
- Storing and Sharing session variables across multiple Intents
- Hitting image Rekognition comparison API limits
- Lex to accept multi languages as input – TranscribeService initialization is not working as it says it’s not constructor.
Accomplishments that I'm proud of
- Sending Camera Feed to Sumerian for processing, customer head movement and identifying user emotions
- Implementing Face Rekognition to confirm Customer identify
- Implementing AWS Translate to translate Lex output up to any one of 15 languages selected by the customer
- End to end integration with Sumerian, Lex, Lambda, Dynamodb, S3 and SES
- Showing data in HTML content dynamically
What I learned
- Capturing camera feed through Sumerian
- Making calls to Face Rekognition and Face Comparison from Sumerian
- How and where to use Rekognition Comparison to avoid throttling
- Implementing Lex in Sumerian without using state machine to control the responses from Lex
- Using AWS Translate to translate Lex response
- Sharing session variables across intents
- How to effectively create Hash Keys for Dynamo DB for querying
What's next for VRPersonalBanker
- Implement transcribe to implement voice to text conversion for accepting input in multi-language
- Implement AWS Comprehend to analyze customer sentiments and generate a report so Banks can improve their services offered.
- Add more functionality to cover all the activities done at Branch through this application
- Incorporate text based user input for the hearing and speech abled customers