Restaurant Menu Bot

menu in Catalan language from restaurant Amar Barcelona.
menu in English from Frenchette restaurant in New York.
menu in Turkish from restaurant Telezzuz in Istanbul.

Inspiration

Being a New Yorker myself, I know most New Yorkers won't settle for disappointing food choices at restaurants. We expect our meals to both surprise us and accommodate our ever-changing flavor preferences and dietary restrictions.
The ability to order from menus in languages other than English greatly enhances the experience of traveling.

What it does

This BOT helps user to explore the MENU items of any restaurant, of which MENU is n PDF format. The pdf MENU is extracted by Python code and fed to the model as part of the prompt.

BOT starts the chat by asking user to upload the MENU of restaurant of her choice.
- The menu pdf file must be in local directory (i.e., in the same directory where python script is).
- Menu can be in English or in other languages.
- Example menus are provided: one menu in Catalan, one menu in Turkish and three menus in English.
- Restaurant menu must be in pdf format.
BOT checks user's dietary restrictions (e.g., no dairy).
User can ask recommendations (e.g., I want to eat a dish which tastes both briny and fruity) or questions about ingredients.
User can ask for changes to dishes (e.g., olive oil instead of butter, extra spicy, steamed not fried).
User can order menu items.
Conversation ends when user confirms the order.
BOT offers a lesser-known fact about one of the ordered items before ending.

See the demo video for 3 different conversations between BOT and user to select items from 3 different restaurant menus.

How we built it

I used the Python SDK for the Gemini API. It is written as single Python script.

In the Python script, automatic function calling is used to build the bot. Functions are defined for learning user's dietary restrictions and ordering from menu. Functions are named and commented appropriately so that the model can interpret them sensibly. They are passed to the model constructor of Gemini API as tools.

The prompt is constructed from the extracted text from the uploaded MENU pdf file and appended to manually written prompt containing instructions and guidance.

The model is configured as a chat loop, which ends when user confirms the order.

Challenges we ran into

As a novice to Gemini and Google development tools, I needed to learn them. But most of my time was spent to figure out Google setup. For example, where to find Google Cloud Project ID, which is required for this hackathon.

Accomplishments that we're proud of

This BOT can internationally add value to restaurants by increasing number of satisfied customers.

I am proud of:

Creating a BOT that is addictive (I use it now before ordering from any restaurant).
Finding and using a feature Gemini API is very good at: The BOT seamlessly uses restaurant menus in different natural languages without programmer's effort.