SeeFood* is the mobile application that finally lets you reverse engineer food by simply taking a picture of it. For those dietary-conscious people that love to eat out, taking a picture of a dish before you dig in will give you a close estimate of its ingredients and nutritional facts.

Many people want to know the nutritional facts of what they eat for various dietary reasons such as diabetes, celiac disease, liver disease, kidney disease, among many others or they are just trying to be conscious about what they eat to lose weight.

The user just whips out their Android phone to take a quick picture of the meal and taps a virtual button to send the image off to the cloud managed by someone with the processing capabilities on par with the likes of Google, if not Google itself. Upon processing the image, the cloud pushes back the nutritional information as a result. Since many people enjoy sharing their meals on popular social networking sites as Google+, users can simply opt to push the image (minus the nutritional facts of course!) to their Google+ updates.

In addition to sending the picture to the cloud to be analyzed, the user can also opt to include her location info, restaurant name, meal name and a short review to add to a crowd sourced database of menu offerings by restaurants. As this database is filled out, future patrons of restaurants in the database can pull up submissions from others to see (on their phones or overlaid on top of the menu via Google Glass) averaged, estimated nutritional data and possible reviews of items on the menu ahead of time to help make informed decisions about what to and what not to order.

Additionally, the app can offer a deluxe feature of returning the ingredients detected and can suggest popular recipes from around the web with similar ingredients to best recreate the dish experienced at home with possible partnerships from the likes of Food Network, etc.


SeeFood will have a two-pronged monetization approach of ad support or subscription fees. A free version of the app can display ads based on the meal submitted to show ads/offers for the current restaurant, similar restaurants, or even grocery stores offering targeted discounts on ingredients for meals rated particularly high by the user. It may be a possible business decision to withhold some functionality from the free version such as the ingredients and suggested recipes for the subscription-based version depending on how well the ad-supported version does monetarily for advertisers.

A subscription-based version will do away with the ads while offering complete access to all services.

Technical details:

This sounds like science fiction but it is theoretically possible given today’s off-the-shelf mobile phones. The process relies heavily on the wonders of computer vision. Firstly, a mobile phone with stereo cameras takes a close picture of a prepared meal. The stereo cameras act as a range camera to determine the depth to all points visible between the two images. Mobile phones with stereo cameras (often referred to as 3D cameras) are currently available on the market offered by companies such as LG and HTC [1]. By 2015, it is estimated that 80% of smartphones will have 3D cameras [2]. With the addition of ultrasound being added to phones by Qualcomm, it is possible that the range sensing accuracy from the cameras could be bolstered by ultrasonic ranging depending on how Qualcomm implements their ultrasound strategy. In parallel to the depth information being calculated, the image is processed to determine discreet objects such as a chicken breast or rice by processing natural features and color blobs. The depth data is applied to the recognized objects to estimate their volume for determining portion size. This can essentially be thought of as a Microsoft Kinect on your phone.

Additionally, it has been shown that smartphones can be used as a basic spectrophotometer [3]. With the added spectrometry data, basic chemical properties can be estimated to refine the results of the detected food as well as possible sauces, ingredients, and additives. For example, an image of a chicken breast with a buttery sauce could have a distinct spectrograph that divulges the addition of the buttery sauce to the chicken breast that would otherwise have been undetected through the image alone without additional exhaustive machine learning.

Now that we know the type of food, its estimated volume (portion size), and any potential sauces, additives, or extra ingredients, it is relatively easy to estimate its nutritional information from open databases on the internet. This is essentially Google Goggles 2.0.

Things that we would love to add in the future:

Further inclusion of an olfactory sensor in the smartphone hardware could increase the accuracy of the detected ingredients. Qualcomm surely has the engineering prowess to develop and add this feature to their future offerings. The olfactory data could even be stored in the Google+ post for in the future when we have olfactory actuators in our smartphones to recreate the smell of the meal that our friends shared with us.

Inclusion of diet logging functionality could make this the de facto standard for diet logging apps. If users prefer to use other existing apps, SeeFood can export nutritional data for consumed meals as an Android Intent or as a flat file spreadsheet that can be uploaded to Google Docs.

This is purely in the idea stage and needs more research from the likes of Google’s and Qualcomm’s superstar engineers to produce a working prototype.

*SeeFood is a working title as it is unclear if it is trademarked already. There is another mobile app that is unclear about whether it is called “SeeTheFood” or “SeeFood”





Application Image by Hani Arif used under the permissive license provided by the owner.

Share this project: