Virtual Intelligence Merchant(Chat-bot)

Inspiration

As business owners, we want to reach out to our customer by any chance. We think about our disability customers who can only ready and write, and we want to provide the best service to our value customers.

We are sometimes getting really busy and if we want to order a "quick" drink, we will need to place orders in advance. In addition, we usually don't download a new app for ordering. What is the quickest way to do so? Well, what if we can place order using our day-to-day messaging? What if we can interact directly with merchants using just our regular text message that we call Virtual Intelligence Merchant and tell them exactly what we want and still be able to look around different menu options?

In addition, what if a merchant wants to reach out to their favorite consumers about their new drinks, new discounts, new seasonal, or just want to stay in touch with their customers ect?

After going through project design thinking, we like to build a Virtual Agent that can be intelligently understand users's text messages to help increasing consumer experiences and help bonding consumer and merchant.

What it does

The Virtual Agent is able to list out different items and categories from NCR Silver API Customers can search , add item to cart , delete item from the cart, place orders, be able to receive delivery option. Virtual Agent can understand what users say by using Natural Language Processing to match to the closest items for the customers After customers have placed their orders, the notification will be sent to merchant using NCR Silver API, the merchant can see on their Ipad who orders and what they order. Virtual Agent can match exactly how many quantities of the orders that customer'd like to order. Virtual Agent can locate the location of the customer and enable the delivery option

How We built it

Java Spring Boot is used to handle all the back-end logic around the chat-bot. We used Java to handle all the HTTP requests to NCR Silver API, we create our custom models to match with NCR Objects. We used Python Flask to handle the integration between Diaglogflow and our Java application , and to handle Geofencing to locate user's location. We trained our NLP with Diaglogflow with different intents, and entities. We used Azure to host our content images. We used Android Rich Messaging as a chat-bot platform (RCS api) Ngrok is using to export our localhost H2 database to store in-memory data

Challenges We ran into

Understanding different APIs are truly challenged. We spent hours just to understand the APIs, design, and think about how to integrate them together. For example, Ordering API from NCR has different data structures that we have, we need to re-configure our data model to match with NCR data model

Problem of hosting images on Azure, we ran into the serious downtime when all of the images from Azure stop working. We are not sure why, but we might think it is because of public-private access

Comprise all the duplicated orders into one order. For example, if the customer orders 4 mocha, we don't want to display 4 different images of mocha. We need to write the new algorithm to comprise them into one with the quantity,

Configured NLP API is extremely difficult and challenged. Dialogflow can send the POST request to the webhook that we set up, but we need to figure out the way to send the message back to Dialogflow. That process took a lot of hours and team effort in order to implement a script that can handle HTTP request from Java App to Dialogflow

Accomplishments that We proud of

Followed Agile development with small tasks, code reviews regularly with every Pull Request , effective collaboration between team members Able to make requests to NCR Silver API Able to create order and show the order details on the Ipad Successfully train a small set of use cases using NLP solution Successfully deliver core features of the app Successfully resolve bugs within limited time Optimized the code to work better with low quality of internet Fun fact: Ability to stay awake with 4 hours of sleep for two days

What We learned

New technologies, Ability to think out of the box Design thinking

What's next for Virtual Intelligent Consumer Agent - ChatBot

We want to be able to integrate the delivery API from NCR to support shipping solutions Train more language model for the chatbot to become better and smarter

Built With

• azure
• dialogflow
• flask
• geofencing
• java
• natural-language-processing
• ncr
• ngrok
• python
• rcs
• spring-boot