AR Food Menu

Inspiration

⌛ Speeding up the process of ordering takeout with the Snapchat group!

📈 +160K Views and 240+ Shares in just 5 Days!

What it does

AR Food Menu ML to identify a fast food establishment's sign or logo and display a preview of their top items.

How we built it

The Machine Learning Model was built using Tensor-flow lite. Training data was scraped from the web using Microsoft azure's cognitive search. This means that adding new establishments to the compatibility list requires a low amount of effort.

Lens Roadmap

In the future we plan on training a MaskCNN which would allow us to draw a bounding box around the logo/sign. Using Snap's AR persistent Note-board we could pin the menu to the sign in the AR world allowing users to glance at many restaurants at once before deciding.

Challenges we ran into

The largest challenge was creating the custom ML from scratch. This was my first time ever using Tensorflow, lens studio, and azure. Getting all the data and manually going through to clean it and improve it, and then training it was a big challenge. All while learning how to use these new tools at the same time.

Accomplishments that we're proud of

We're very proud of the custom model. It is truly something that is not available anywhere else on the internet and made to work with lens studio.

What we learned

Learned that there is a 4K+ user base in India just after 3 days. We have to tailor the experience to support popular spots there. That lens studio has a wide array of AR tools. Learned how to clean and scrape data with azure cognitive search and then feed into TFlite to make a lightweight fast consistent model.

Built With

• azure
• javascript
• python
• tensorflow