What's on the menu?

Visualize menu items in realtime using computer vision

Comment

Inspiration

When you go to a restaurant, you often can choose from a lot of good choices. Sometimes it can be confusing to navigate, and you might not know what some exotic dishes even mean! By using the crowdsourced images from multiple social networking websites, What's on the menu? is able to aid in deciding what you want to order, based on appearance, size of dish, presentation, all possible from past photos of the menu item. All images are from sites such as Yelp, Instagram, Pinterest, and more to find as many relevant photos for that dish from the restaurant you are at currently.

What it does

What's on the menu? uses your phone's camera and computer vision to read and interpret menu text in real time and dynamically looks up what the dish looks like in the restaurant you're eating using google custom search api resolved through stdlib, searching through all popular image sharing platforms such as Yelp, Instagram, Pinterest, and more. In addition, it utilizes GPS location tracking and the Google Places API to further fine-tune the results. It searches using this data to try to get the most accurate images based on the user's location and the text that it read from the menu.

How we built it

We built it the app in Java using Android Studio. The API it uses are Google Places API to get a more accurate reading of where the user is eating, Google Custom Search API to search for the dish images at a certain restaurant, Google Cloud for computer vision in text recognition, and stdlib to streamline our API calls.

Challenges we ran into

Some challenges were making the app get the images to show up in a nice layout and getting the API calls to work how we wanted them to. It was also challenging to bundle various libraries and services together and make sure they don't conflict with each other.

Accomplishments that we're proud of

We're proud that it is able to give photos based off of the custom search api results in real time, much like how Google Translate works with very little overhead. We tried the app on menus from restaurants around Princeton, and we had very good results! We learned what "clams casino" was from Blue Point Grill and had a great dinner.

What we learned

By leveraging existing technologies and combining them in new ways we can get surprising results. Sometimes google image search is not 100% accurate even with a lot of information. Lemon chicken.

What's next for What's on the menu?

Possibly AR for more dimensions of photos.

Built With

Share this project:

Updates