MenuMate

MenuMate makes restaurant menus accessible to everyone and helps encourage you to experience new cultures.

Comment

Inspiration

Sharing a meal has always been a great way to bond with friends, family, and your community. However, elderly individuals or people with impaired abilities (e.g., vision impairment, dyslexia) might feel uncomfortable dining at restaurants. That’s where we came up with the idea of _ MenuMate _ —an app designed to empower and ensure everyone can enjoy a wonderful dining experience!

What it does

_ MenuMate _ is an application that allows users to receive translated and simplified menus with ingredient annotations by simply uploading a restaurant menu. It can also summarize the menu, providing users with an overview of the restaurant (e.g., its style and price range) and recommend dishes based on their dietary restrictions, culture, allergies, and preferences. Additionally, it can suggest meals similar to the user's cultural cuisine. It is focused on making this experience accessible to everyone with easy to navigate UI, dyslexia-friendly fonts, and STT and TTS for visual or motor impairment.

How we built it

  • Frontend: React Native
  • Web Design: Figma
  • Backend: Flask
  • AI Integration: Python using Gemini API
  • Not ChatGPT :)

Challenges we ran into

  • Integrating various functionalities into the app (e.g., capturing pictures, converting to binary/URI, and sending them to the backend)
  • Setting up the server using Flask and deploying with Expo
  • Ensuring communication between the frontend and backend to trigger functions upon button clicks

Accomplishments that we're proud of

  • App is deployed on web using React.js and Flask
  • Built a UI on mobile with React Native and Figma
  • Allows user to upload menu pictures and have full functionality (simplified menu, summary, suggestions)

What we learned

  • The workflow of building a full-stack application
  • Utilise Gemini API and its capabilities
  • Setting up web server with Flask

What's next for MenuMate

  • Enable user to take and save menu pictures locally
  • Deploy the app on mobile devices
  • Expand recommendation features to include personal preferences, cost-effective dining suggestions, and sustainability index recommendations
  • Speech-to-text feature, which allows user to input their preferences and settings easier
Share this project:

Updates