Category of Submission

Brand Engagement and Retail


When thinking about how hosts could be utilized in retail, we thought about the extensive beer sections in grocery stores and how it would be nice to be able to casually and naturally have a quick conversation to find beers that you like. Especially when visiting new places, it’s impossible to know what the best local beers are. If you ever go to the grocery store, you find people just roaming the aisle for extended periods of time, trying to read each beer label to find one that looks interesting. If you ever grab an employee for help, you generally supply them with what types of beer you like (IPA, stout, pilsner, …), what characteristics you prefer (hoppy, malty, …), and then if you’re looking for local beers or not. As a result, we wanted to explore the idea of using Sumerian for a kiosk-style experience, to gather this information as if you were speaking with a beer expert right there in the store.

The first thing we tackled when approaching this topic was how to initiate the experience without requiring a standard touch or click like normal kiosks without a host. We simulated the hardware portion with a Raspberry Pi. Using a PIR motion sensor to detect someone walking up to the kiosk, we then emit a message over web sockets using This communication is intercepted by the Sumerian scene and initiates the chatbot.

The next thing we wanted to figure out was how to let the user have a conversation with the Sumerian host without having to use key presses or other methods to signify the start and end of a speech input. We used the Web Speech API which has built-in support for automatic speech start / end detection. Once these events are received in our custom scripts, we emit SystemBus messages to communicate with the host chatbot’s state machine and have them hooked into the normal chatbot flow with some slight updates.

In order to detect the end of the conversation and respond accordingly, we added a listener to the Lex bot. Messages are received from Lex throughout the conversation for each input / output but we listen for the state of the chatbot to be “Fulfilled” and then send the result over a postMessage. A custom HTML entity with a React / Redux app is waiting for this postMessage and grabs those parameters, sends them to an API, and displays results.

Designated Device

Chromium-based touch screen kiosk (or desktop Chrome browser for demo purposes due to limited support for the Web Speech API).

Additional Notes

  1. This demo MUST be viewed in Chrome.
  2. Since part of our hack was focused on a physical hardware aspect, specifically a Raspberry Pi, we added a hook into the Sumerian scene demo for the space bar to kick off the experience, rather than waiting for a socket message from the Pi. For demo purposes, please use the space bar to start the demo but you do not need to use the space bar to continue to interface with the chatbot.
  3. Make sure your volume is turned on.
  4. Since the focus was on the hardware and chatbot rather than the depth of the resulting API utilization, we are not filtering responses based on your location for local beers. In order to simulate this, we are automatically applying a location filter for California, Washington, and Oregon, as if you were in a Seattle, WA store when using the Beer Concierge.


In addition to the Sumerian scene demo, here are the repos for the hardware server and beer results app:

RaspberryPi motion sensor repo

Beer results React app repo

Share this project: