What if you could simply order and pay for something without having to carry any cash, credit cards, or smartphone with you - just using your voice and face, but still in a secure fashion?

What it does

Our vending machine listens to your order. Once it positively identified you using both your voice and face (for enhanced security) your order is processed and your desired candy is dispensed. Simple as that.

In case of an unspecific order, e.g., "give me some candy, please", advanced face detection technology is used to automatically decide which type of candy suits you best, depending on your current mood. Happy? Order with a smile!

Additionally, our system supports you in reaching your health goals using your personal diet plans. If you still want that extra chocolate bar, we got you covered - simply use the magic word (sudo) and your wish will be fulfilled!

How we built it

The hardware is based on a Raspberry Pi micro computer with sensors, to record your voice and take a photo of your face, and actors, to fulfill your order. In detail, we use industry-standard metal helices attached to servos to dispense the desired candy. A touchscreen display provides further feedback to you and is currently used to initiate the order process.

Our software is written in Python and makes use of the following great APIs:

  • Microsoft Speaker Recognition API to identify you using just your voice,
  • Microsoft Face API to validate your identity using a photo of your face,
  • Google Speech API to transcribe your spoken order into machine-understandable text, and
  • Google Vision API for advanced face detection capabilities, here: to detect your mood.

Challenges we ran into

A big challenge was the final assembly and adjustment of our hardware. Additionally, integrating and fine-tuning the used APIs was partially challenging. Finally, building a GUI for our Raspberry Pi setup was also a novel experience and a bit challenging.

Accomplishments that we're proud of

It works! You can simply say which candy you'd like to have, take a look into the camera, and our system identifies you and fulfills your order.

Our advanced features provide an even better customer experience by helping you reach your health goals or provide the best candy depending on your current sentiment.

What we learned

  • Building an app for Raspberry Pi using different hardware components, especially camera, microphone, and servos, and
  • Using and integrating a set of novel APIs from different companies to build a functional and enjoyable user experience.

What's next for Buy Candy with a Smile

  • General availability (vending machines everywhere!),
  • Automate registration process,
  • Payment integration, be automatically charged when you order, and
  • Automatic order start detection ("always listening")

Built With

Share this project: