Get The Formuoli

Remove the hassle of cooking, get all you need in this WhatsApp bot.

Comment

💡 Inspiration

See a dish that you don't know the name of but would like to learn to create? Take a picture and get the recipe, ingredients and video tutorial in an instant! Get The Formuoli is integrated with an everyday social platform, Whatsapp, to provide users easy access to it. All without needing to download external apps, it's so quick and easy you can even get the recipe before you can say "Ravioli Ravioli give me the Formuoli"!

🔍 What it does

It gives you a quick shortcut, jumping straight from taking an image to having all the information about it! Not only is it able to identify your dish, it is able to give you the recipe, serve you the instructions to make it, and the ingredients needed to recreate the dish you are searching for! You would also be able to view a tutorial video on how to cook your dish, all at your fingertip!

Potential use cases

  1. For example if you see a meal on a menu you'd like to try to make at home, or see someone post a photo of a meal on social media, you can take a picture or screenshot the image and simply send it to our WhatsApp chatbot and get all the information you need right away without any further effort required!

  2. Maybe you are at the grocery store and you thought of making an apple pie. Now, there is no need to panic and search up what ingredients are needed. With Get The Formuoli, you can get the ingredients for any dish in 1 short message.

🔧 How we built it

  • Using Twilio to create a WhatsApp bot to get messages and images, specifically the image of the dish
  • Using GCP to host our bot's backend to collect the messages and images from the user, process the data and send a reply back to the user with the recipe information
  • Using ngrok to create a tunnel from localhost so that our WhatsApp bot could read images and messages
  • Using Imgbb free API to host the image on a publically available url
  • Using SerpAPI to perform a reverse Google Image search to find name of the dish using the image at the publically available url
  • With the name of the dish, we can now use a recipe API, TheMealDB, to find the ingredients and instructions to make the dish, as well as links to a video tutorial
  • Finally we return the neatly parsed recipe information to the user via WhatsApp, streamlining the entire process
  • We also added a landing page to introduce new users to our Chatbot, hosted with Vercel
  • We also created our domain name using Domain.com

🏃‍♂️ Challenges we ran into

  • Rate limits on the APIs that we want to test that we needed to keep mindful of
  • It was our first time using Twilio, and not only did we need to figure out how to get the messaging via WhatsApp to work, but we also needed to figure out how to save and process pictures sent to our chatbot via WhatsApp
  • It was also our first time using Imgbb and SerpApi
  • While SerpApi did have the ability to perform a reverse image search, the recipe search function did not provide the information we needed, so we needed to connect an additional API to get the required information.
  • We had some trouble getting the Flask app hosted with GCP to work with Twilio at first. Our quick fix was to create a tunnel to our localhost with ngrok.

🏅 Accomplishments that we're proud of

  • Coming up with an idea that actually has value added
  • Integrating multiple APIs into our application to get the information we need
  • Quickly setting up the front end landing page to introduce users to our Chatbot, and even managing to host it with Vercel!
  • Managing to get Twilio to work and create a cool WhatsApp bot that takes in images and messages!

🧠 What we learned

  • Learned how to use Twilio and return a reply (recipe information) from an image or message
  • Learned about ports, tunnel, port forwarding
  • Learn more about the capabilities of Google Cloud
  • Learned how to write a Flask application for our Twilio WhatsApp bot
  • Learned how to use the various new APIs
  • Learned how to use .pug files for the development of our landing page

⏭️ What's next for Get The Formuoli

  • Buy a phone number to make the Twilio client functional in the long run
  • Find a bigger database of recipe to get more variation
  • Website to perform the search via web (ie 1 step upload photo and get recipe)
  • More commands for our chatbot to allow it to provide the user with more information for the cost of ingredients and options to quickly order all the required ingredients(online).
  • Alternate options for the chatbot to should you where you could buy the dish if you don't want to make it yourself

Built With

Share this project:

Updates