Inspiration

As a care giver, I have been hyper-sensitive about my contacts in this COVID-19 world. In looking for ways to use technology to improve on processes, I considered my local small businesses. Order taking requires risky face-to-face dwell time that can be limited with the right solution.

What it does

Voice Order implements an automated contactless order taking and payment service using a voice-enabled kiosk and Square services for a catalog of items, order logging, payments, and loyalty rewards. Voice Order uses wit.ai to provide a natural language experience for placing an order. The kiosk runs on an iPad and communicates with a Brother printer for the kitchen or fulfillment area to prepare the order. There is a companion Voice Pay app that interfaces over bluetooth with the kiosk Voice Order app. The Voice Pay app identifies you to the service, enables contactless payment via a credit card with Square, or Apple Pay, and provides order confirmation.

The merchant logs into the Voice Order app and can set up their service by identifying their printer and reviewing their menu. They can also review recent orders. When ready, they enter kiosk mode in the app to take orders. A user launches the Voice Pay app and brings their phone near the kiosk to sync. They are then presented with a menu. As they speak to the kiosk, it processes their speech on wit.ai to assemble their order and shows the total cost. The Voice Order app automatically applies any earned rewards, such as a free item. The user can add a tip, and finish their order. The kiosk then prompts to make a payment on their Voice Pay app. When that's done, the kiosk shows confirmation and prints the order on the prep area Brother printer. The Voice Pay app shows their complete order with an order number so they can claim the order when it's brought forward.

The system is then ready for the next customer.

The whole system uses the Square service to host the order item catalog, to record orders, take payments, track customer loyalty, etc.

The wit.ai service is used to process what the user says in a natural manner to compose their order, and handle tasks such as adding a tip, and confirming the order.

How I built it

I created 2 iOS apps in Swift - Voice Order and Voice Pay. I used the Brother SDK to interface to the printer. The Voice Order app uses iOS voice recognition technology to create the text sent to the wit.ai service. The Voice Order app interfaces with wit.ai using their API, receiving back intents, entities and related data. It interprets this information to formulate the order and take associated actions. Both apps use bluetooth technology to synchronize information between them. They both use the Square SDK to interface with the payments service. The Voice Order app further interfaces with the Square service over their REST API to manage the menu (via a catalog), order taking, loyalty rewards, and payment fulfillment.

Challenges I ran into

There were numerous challenges in building this service from scratch. First, getting the voice interface working so that I could associate intents and entities from the wit.ai service with a catalog of menu items from the Square backend took a lot of experimentation and testing. Second was interfacing to the Brother printers, which took a while to learn, but I had some help from Brother support. Third was implementing the bluetooth sync interface between the two apps. That was very hard to debug any issues. Finally, interfacing to the Square service took a while to figure out, debug, and test. The payment SDK was somewhat easier to integrate, and there was some good example code. But the REST API required a lot of experimentation with the API explorer, trial and error in the code implementation, and proper sequencing of the calls to effect the transactions.

Accomplishments that I'm proud of

I'm very happy with the progress I was able to make in just a few weeks. While it still needs work before deployment, it's working well as a demonstration. There were a lot of pieces that all had to work together to even get to that stage.

What I learned

I learned a great deal about each of the various pieces. How to implement a voice-controlled interface, how to talk to Brother printers, how to sync information over bluetooth, and how to work with the Square service with their SDK and REST API.

What's next for Voice Order

I want to identify a local merchant where I can field test the service. There is still work to do for account creation, merchant definition, menu editing, and some other components before Voice Order is ready for deployment.

Built With

Share this project:

Updates