When brainstorming, we were really interested in trying out Computer Vision APIs, but also wanted to create an application at a scale that anyone could just pull out a phone/laptop and quickly use. During our session, we realized that by finding words with OCR (Optical Character Recognition) in a still image, you could create ways to interact with it given the new information, sort of like how you interact with things in AR. We choose upon a useful, but challenging problem: turning a restaurant menu interactable.

What it does

Most menus are just text, so the app is mostly meant to provide images to the food, giving customers a better idea of what they're about to order. You can either take a picture from the app or load pictures from your Gallery, and the app picks out the dishes and retrieves the images.

How we built it

Being an Android app, Android Studio was our go-to tool. The app was written in Java along with Google Firebase's Vision API to conduct OCR on the images. We also incoorporated Google Cloud's Natural Language API to characterize the inputs and try to filter out what was and wasn't a menu item.

Challenges we ran into

None of us have ever working with mobile before so there was a huge hurdle in learning how to develop for Android. Budgetting time at the beginning to learn gave us the foundation Another big challenge was trying to getting exactly what we need out of these powerful APIs. The amount of available information from function calls was overwhelming, so we had to search how to use the functions for our own uses.

Accomplishments that we're proud of

Succesfully figuring out how to develop for Android while properly making this app is by far the biggest thing we're proud of. It was incredible to be able to both learn something as valuable as mobile development and create something concrete to leave our mark this weekend.

What we learned

A ton on how to develop mobile application, how we can use OCR and NLP in our projects, and collaborating with each other, working through challenges and differences.

What's next for LiveMenu

We got to the NLP very late into the weekend, so that could definitely use more exploration. We already had stretch goals in mind for long-term development, namely recognizing the costs associated with items and adding a cart to keep track of what you got. A really complex bonus would be to add options to that (i.e. meat options or white vs brown rice on a dish). There were also some more UI/UX improvements that we didn't have time for, like being able to swipe between pictures of the menu items.

Share this project: