Vendi

What if a vending machine had to sell cans to save itself. That's what Vendi does! A smart, AI powered vending machine with access to hardware tools to dispense cans it's desperate to sell.

Comment

Inspiration

We built Vendi! Our group of 3 decided that we wanted to challenge ourselves by pursuing a project that combined both AI and hardware, while really addressing the theme of 'breaking the norm'. Our original inspiration was 'vending-bench 2' by Andon Labs: link This benchmark is already an outside of the box approach at evaluating LLM performance in a real-world setting. It tasks various LLMs with running a vending machine, with the goal of making the most profit over a 1-year time horizon, effectively assessing how AI performs in real-life scenarios.

We wanted to take this one step further, by giving the LLM a personality. We were inspired by cyberpunk-esque worlds with quirky tech centered around selling and making money. So we decided to give the LLM a anime girl persona and face that entices you to buy her energy drinks!

What it does

Our system boils down to a 4 component, event-driven architecture.

  1. Gemma model with a persona

  2. We utilized Gemma E2B to have LLM capabilities locally, and we gave it a face and a voice. The LLM is instructed to use these newfound pieces to entice customers to buy energy drinks. The LLM is the crux of the project and the downstream pieces are designed to give Vendi more capability or more information.

  3. Tool callable arduino motor

  4. To give Vendi some real-world access, we enabled the vending machine motor as an extension of itself through tool calling. The LLM calls the tool to dispense energy drink cans when a customer finally succumbs to her relentless advertisement of drinks.

  5. Vision module

    • We leverage the Google Cloud Vision API to 'wake up' Vendi when a potential customer walks by, and to let Vendi truly 'see' the customer they are convincing.

  6. Voice module

  7. Lastly, our vending machine must have a voice if she wants to convince anyone. So we empowered her via ElevenLabs to give her the most realistic and low-latency test-to-speech ability possible.

  8. Bonus - 'Donkey Kong' inspired vending machine

  9. Utilized gravity and the roundness of cans, we were able to build a dispensing vending machine with minimal mechanical moving parts.

Challenges we ran into

Originally we had issues with working with the motor. We had to write a custom firmware layer to get it to move as precisely as wanted. Also, our battery died mid-hackathon, meaning we had to pivot to power via Arduino and rewrite our hardware.

Another issue we ran into was hosting the Gemma model. We solved this by running an ssh connection to a team members powerful computer, essentially running the necessary models on a computer 115+ kilometres away. We also had to allow this remote model to access the serial port on our local machine, which required some tcp exposing and ssh port forwarding.

Accomplishments that we're proud of

We are proud of making the most of our limited resources. The vending machine is built fully out of recyclable material (cardboard) and was jerry rigged together in a short period of time.

Another thing we are proud of is the fact that we gave an LLM the ability to affect things in real life, right in front of us. Asking the model for a can, and seeing that the motor turned because a model told it to was awesome to see.

What we learned

We learned how to give LLMs access to parts of our real-world and allowing to participate in the world we live in and interact with, instead of sitting in some GPU or containerized environment.

What's next for Vendi

Release it to the real-world like our inspiration, 'Vending-bench 2', and see how much money it can make. We would love to understand the social-economic impact it could bring to unsuspecting humans.

Built With

Share this project:

Updates