dishcovery

Maximizing cultural experiences through food discovery

Comment

Inspiration

Trying out new cuisines should be an easy and enjoyable way to learn more about other cultures and their traditions. However, it can be daunting to browse a restaurant menu full of dishes you've never seen before, with names you can't pronounce. You may not be able to imagine how a dish may look or the menu may not come with an adequate description. With Dishcovery, we aim to empower people to experience more diverse cuisines.

What it does

Dishcovery enhances the menu-browsing and ordering experience - providing images, pronunciations, and descriptions of each dish. Just take a photo of the menu with your device's camera, or upload an image directly, and the application will parse the menu. Each dish is matched with an appropriate image and descriptions which the user can then browse through. Tap on the sound/volume icon and the name of the dish will play out loud, allowing users to learn the pronunciation and order with confidence.

How we built it

The application is built with HTML, CSS, and JavaScript. We used the frameworks materializeCSS and jQuery to help with styling and boiler-plate code. Our main capabilities are powered through Microsoft Azure APIs. The Cognitive Services' Computer Vision API enables us to extract the names of each dish (either printed or hand-written) from the menus. To retrieve the appropriate images and descriptions of each food item, we use the Bing Image Search and Web Search APIs.

Challenges we ran into

As we were developing and testing, the frequency at which we were querying some of the Azure APIs would hit the rate limits for the free tier of the service, and repeated identical calls would be blocked. We also found that when testing with perfect images, the Computer Vision API performed with extreme accuracy, but images captured using a phone camera or webcam were less predictable. The errors were minor, but could result in the wrong images and descriptions being generated. If we had more time, Azure's custom image search/a more customized approach would likely improve the quality and consistency of our results.

Accomplishments that we're proud of

Our team is proud of being able to build a functioning product, accurately matching a wide range of food items with the correct image with the help of Azure's APIs. Each of us learned a lot building Dishcovery (You could say we "dishcovered" new skills) and refined our existing capabilities.

What we learned

We learned that the Azure Cognitive Services APIs are very flexible, and can be applied to specific use-cases, either in isolation or when combined with other Azure APIs.

What's next for dishcovery

Some features we could build out to improve dishcovery include:

  • Adding Azure Bing Spell Check to improve the accuracy of parsing camera-captured images, many of the imperfections in the OCR for captured images are minor and can be corrected.
  • Integration with Yelp or other food-review services to use customer-submitted images of dishes from the specific restaurant that the user is dining at.
  • Personalizing the experience by recommending dishes based on the user's order history and preferences.
Share this project:

Updates

Submission history