voice enabled kiosk

• 

  talking to kiosk

Inspiration

What if the user doesn't know english to understand the kiosk, we have 30% hispanic and asians in USA. Also kiosk at Taco Bell, has huge list, I need to drill down to get to my item. What if I have an query about an item (dont know what queso is). With advances in AI we can enable kiosk to respond to voice queries, so that consumer can appreciate visual information and converse as if he was talking to cashier for the on-site consumer experience.

What it does

We boldly call it build your own Alexa, seller can brand it the way they want and its totally private. Consumer can talk to kiosk, as shown in the demo video.

How we built it

By embedding our java script widget or Android sdk, we provide voice activation to the kiosk. Consumer start talking, we do the dialog management also visual information based on the item library from square.

Challenges we ran into

LLM hallucinate, makes up the modifiers that doesn't exist. We need to build pipelines for supervised fine tuning for accuracy.

Accomplishments that we're proud of

What we learned

voice bots are harder than text only, chatGPT plugin. There shouldn't be excess delays. For example we dont know if consumer has said one cheeseburger please or cheeseburger with cheddar and onions. Having a default timeout would create unnecessary pause.

What's next for voice enabled kiosk

Drive through automation is next frontier.

Built With

• gpt-4
• python
• websockets
• whisper