MealCheck Vision

Take a photo of your Korean meal and see if it fits your health conditions. It checks with you when it isn't sure what a dish is.

Comment

Inspiration

Most older people in Korea have at least one chronic condition, and a lot of them have two or three (86% and 64% of people over 65, per the 2023 national survey). Doctors tell them to eat less salt, but that doesn't help much when there's rice, a stew and five side dishes on the table. It gets harder with more than one condition. Potassium is fine for high blood pressure but bad for kidney disease, so the same side dish can be good for one and risky for the other.

I'd already built MealCheck, a web app that checks a meal against 30 conditions using fixed rules. You had to type in every dish by hand, though, and nobody wants to do that three times a day. So I wanted a camera that could look at a whole Korean table and tell me when it wasn't sure.

What it does

You pick your conditions and take a photo of your meal. The server finds each dish, names it, and draws boxes on the photo. If it's confident, the box is green. If not, it's orange and you pick from three guesses (or "none of these"). You can't get a verdict until every orange dish is answered. After that, MealCheck's rule engine gives a Good / Caution / Avoid for each condition, with the actual numbers and the guideline it used.

The model only names the food. It never decides whether the meal is healthy for you. That part is always the rules.

How I built it

Everything runs on AWS Lambda on an arm64 (Graviton) instance, Python 3.12, with OpenCV 5.0.0.

  • Preprocessing: decode, gray-world white balance, CLAHE on the L channel. Home and restaurant lighting is very warm and red stews all start to look the same.
  • Finding dishes: Korean tables are mostly round bowls seen from above, so I use HoughCircles, plus Canny edges and contours for plates. I throw out shapes that aren't convex (usually just bits of food texture), merge overlaps with NMSBoxes, and drop boxes that sit inside a bigger one.
  • Naming dishes: a MobileNetV3 I trained earlier on AI Hub Korean food photos (247 dishes), run through cv.dnn.
  • Deciding what to do: if the top guess is 60% or more, it's accepted. If not, the server takes another look with three new crops (wider, tighter, contrast-boosted, flipped), averages them, and decides again. Under 12% it drops the region as "not food". Two boxes with the same label that overlap get merged. Anything else goes to the user. Every step is saved and you can open it under "Agent trace".
  • Logging: one line per request in CloudWatch with timings and decisions. The photo itself is never saved.

Challenges

My first deploy crashed right away. The model in MealCheck is int8-quantized for the browser, and OpenCV can't load that format. I wrote a script that converts the 64 quantized layers back to normal float layers, then checked it against the original (29 of 32 test inputs gave the same top answer).

The first version of dish finding was also way too eager. It cut a single bowl into about eight pieces, so the "confident" answers were only right about half the time. Adding the convexity filter, removing boxes inside boxes and the wider crop got that up to 76.5%.

Results

I tested on 42 public photos from Wikimedia Commons (38 single dishes, 4 full tables). These are pretty different from the training photos, so the raw numbers are lower than in training.

  • The right dish was in the options shown to the user 73.7% of the time.
  • When the system accepted a dish on its own, it was right 13 out of 17 times (76.5%).
  • Of the 20 times it got the dish wrong, 14 were sent to the user to check instead of being accepted.
  • Plain top-1 accuracy was 47.4% (52.6% if you just classify the whole photo). On its own validation set the model gets 83.6% top-1.
  • Median server time was 1.37 seconds per photo.

Red stews (kimchi jjigae, sundubu, yukgaejang) still get mixed up, and flat plates on a crowded table are often missed. Those are the next things to fix.

Privacy and safety

This isn't a medical device and the app says so. Photos stay in memory and are thrown away after the request. Your conditions and results never leave your browser.

What's next

Train a proper detector on full-table photos, estimate portion size from how full the bowl is, get a dietitian to review the thresholds, and test it with older people at a local community center.

Built With

Share this project:

Updates

Submission history