Skyler - The Skyward Bank Voice Assistant

• 

  Skyler Voicebot Flow (01)

• 

  Skyler Voicebot Flow (02)

• 

  Application Sequence Diagram

• 

  Whiteboard Session

• 

  Skyward Bank Logo

Inspiration

Skyler is a voice based virtual assistant for Skyward bank (Note: Skyward Bank is a fictional bank to avoid any conflicts due the usage of real bank). Having a voice bot in the contact center ecosystem is a huge plus. With Skyler, we have integrated Voiceflow with Twilio telephony platform such that bots created in Voiceflow can be invoked from Twilio platform. We have added cherry on the top by providing channel hand-off which means transitioning from voice channel to WhatsApp channel for the same conversation - making this experience multi-channel.

What it does

Skyler can help you with queries related to loans, credit cards, account, check ordering, mobile check deposit, etc. Apart from interacting with the customer on voice channel, Skyler is also capable of smoothly transitioning from voice channel to WhatsApp channel allowing user to use WhatsApp to perform actions for which WhatsApp may be more convenient. This means that the interaction starts with the voicebot on a phone channel. During the course of interaction, caller is offered to continue this conversation on WhatsApp for one of the complex usecase (e.g. Address change). When the conversation is handed off from IVR/voicebot to WhatsApp, phone call is terminated, data collected in voicebot is sent to WhatsApp and is used to drive the conversation from there. In a nutshell, Skyler voice assistant is the following integrations:

1. Twilio Voice Platform
2. WhatsApp Channel
3. Voiceflow APIs
4. BerriAI APIs
5. OpenAI GPT API
6. OpenAI Whisper API

How we built it

We used several tools and technologies to build Skyler. Here is our tech stack:

1. Voiceflow Platform
2. BerriAI APIs (create-app and query-app) to train model on specific domains like credit card, checks, loans etc. BerriAI playground link for reference here link
3. OpenAI Whisper API (For transcription - Speech to text)
4. OpenAI GPT API (completions API)
5. Twilio Voice Platform
6. Twilio SDK
7. Java 11
8. SparkJava framework
9. Springboot
10. ngrok

Challenges we ran into

1. To achieve the interaction hand-off from phone to WhatsApp, we needed a Voiceflow API to trigger WhatsApp message on the given phone number. Voiceflow does not provide such API yet. Also as per instructions provided here link, we need Twilio paid account and we have a free trial account. As a workaround, we had to extract the API from browser's developer console and use it. We were able to trigger the WhatsApp message to the caller programatically with the limitation that we can trigger WhatsApp message only to a fixed number.
2. We have no prior experience in developing using Voiceflow, so it was bit challenging to develop two flows in a short duration for both voice and WhatsApp channel. But we were able to complete it.
3. We initially planned to use Cerebrium for Whisper API as we have a good amount of credits to utilise, but while testing we noticed that there was an intermittent issue with their API, so we had to switch to OpenAI's Whisper API. This needed a code change in the backend code.

Accomplishments that we're proud of

1. Learning about developing flows using Voiceflow because we had never worked on any of these.
2. Using multiple LLMs (OpenAI ChatGPT and BerriAI GPT) in the same application for different purpose.
3. Completing the project on/before time.

What we learned

1. Conversation flow development using Voiceflow platform.
2. Integrating Voiceflow conversations as an API with Twilio and driving the entire conversation from Voiceflow.

What's next for Skyler - Skyward Bank Bot

1. Skyler can be integrated with several other channels like webchat, email, SMS making it omni-channel.
2. Skyler can be trained in other domains like payments, Forex, retirement plans, etc. along with many other banking related domains. Currently, its training is limited.
3. To overcome the challenge mentioned above about triggering WhatsApp message programatically, we can integrate with Meta's WhatsApp APIs so that we can trigger WhatsApp message to any valid number provided.

Built With

• chatgpt
• gpt
• java11
• ngrok
• openai
• sparkjava
• springboot
• twilio
• twiml
• voiceflow