Inspiration
I work in a restaurant and continually I run into the same problems every shift: "I have said allergy", "I cant read this", "What does this mean?", people who speak and read in a different language, and all these problems that hinder making their experience as smooth as possible
What it does
An app that takes a picture of a menu filters the allergies, translates 5 languages, describes unknown dishes to the user (taste, texture, what to expect) in a conversational AI voice with dictation.
How we built it
The frontend is built as a React web app so that anyone can access the app from any device. The backend is built with Flask and python.
Challenges we ran into
Gemini quota and outages. 503 "high demand" errors, then 429 "quota exceeded": gemini-flash-latest only allowed 20 requests a day, and every retry counted toward that.
Accomplishments that we're proud of
Translation in 5 languages with no extra AI requests, English, Spanish, French, Chinese and Arabic, including Arabic laid out right to left. Menus are translated in the same request that reads the photo, and the printed dish names are kept so you can order. And then we have voice-first app that works end to end on a real phone: scan a menu, then Read, Describe and Ask by voice, with answers read aloud. It runs on your iPhone through ngrok.
What we learned
The working with AI aspect and how specific you have to be with prompting it when we asked it to describe a certain dish. There was a "work once" bug that I had to figure out that was stimulating. Phones are stricter than browsers so some permissions aren't remembered the same on either. One reusable function beats duplicated code.
What's next for menu-reader
Full app translation for buttons and menu instead of only the menu, a IOS and android app versions instead of it being on the web, a more natural AI voice, have a build my profile feature that remembers each user's allergies, and deployment.
Log in or sign up for Devpost to join the conversation.