camera on your plate :P
There are more than 168,375,343 posts on Instagram for #food and 76,239,441 posts for #foodporn and these numbers will grow in a mere matter of minutes everyday. Advances in AI has enabled us to figure out intricate details about our images. Publicly accessible APIs have made it easier to scrape data and not worry about training complex models. Even though all these make our life simpler, there still remains the painful task of splitting bills. Splitting bills is a long and tedious task and there wasn't a way around it until now. With our application, you don't have to worry about splitting bills.
What it does
It's a novel solution which leverages the images you upload and the tags you already provide on Facebook, Instagram or social media sites to split your bills wisely among your friends. With the power of AI, we can figure out what are all the contents of a dish just by looking at an image and the restaurant menu. Once we have this, we just use the Splitwise API to update the bill. It’s a clean and easy solution to a messy problem which shouldn’t take more than the time required to upload an image
How we built it
As food pictures are uploaded to our system, the items in them are recognized using Clarifai API. Using the menu which we scraped using bing-search api and pulled out the menus using beautiful soup, we get the price of that item and then splitwise APIs are used to incorporate the transactions involved among the group of people tagged on the original post of those photos.
We used Flask in Python to build a restful API which provides the flow for splitwise APIs. We host the restful API on azure.
Challenges we ran into
Figuring out how to use Splitwise APIs took a lot of our time. Another challenge was to find the menus as there aren’t any readily available API, for fetching the item prices. Combining the front and back end was more painful than we expected.
Accomplishments that we're proud of
Scraped menu using beautifulsoup and created a menu repo. Finding well pre trained models/apis to find out what the images contain Successfully combined a plethora of API (splitwise, facebook-graph, bing-search api)
What we learned
We learnt how to scrape dynamically loaded content from Bing in order to get menu data. We realised how powerful and flexible the Facebook Graph API is. We all conclude that Azure, while powerful, was not that easy to start with and needs to do a better job with the documentation.
What's next for Snap-N-Split
We want the experience to be as frictionless as possible. Webhooks api for facebook will allow us to listen to users updates. As soon as someone checks in at a restaurant, uploads pictures of the food and tags their friends, we can run our algorithms in the background and alert the user when the bill is ready. Using Facebook payments or Venmo API we can go a step further and clear the balance due for the outing. We would also like to solve the dearth of good APIs for getting restaurant menus.