Review food guesses, rename labels, and edit outlines before saving a plate.
Local PlateBack model vs. hosted models: mask IoU on a 12-image public-data pilot.
UVA dining dashboard: recurring leftovers and the plates behind them. Illustrative sample data.
Upload a plate photo or use the laptop camera to start a scan.

Inspiration

A plate coming back with rice left on it could mean almost anything. The same pattern across hundreds of returns is worth a closer look. PlateBack started with a question: could a dining hall use those patterns to make the next meal better?

The intended deployment is at the plate collection area in dining halls such as UVA's O-Hill, Newcomb, and Runk. As students return their plates, a fixed camera would capture each plate and PlateBack would analyze the leftovers automatically. Students would follow their usual routine; dining support staff would get a running view of what keeps coming back.

A custom model that scored above Opus 5.5

PlateBack's locally trained food segmentation model scored 54.12% mean mask IoU versus Claude Opus 5.5's 53.75% on the same 12-image, four-food benchmark. It also scored above Grok 4.7 (36.1%) and Claude Sonnet 5.5 (25.5%) in that comparison.

What it does

The goal is an automatic plate-return analytics station for dining support staff. A camera at the collection area would feed plate images into the model, aggregate the foods left across each meal service, and give staff a way to inspect questionable detections. That continuous camera capture and automatic processing are the planned deployment, not a feature already running in a UVA dining hall.

The dining dashboard lets staff compare similar meals and inspect recurring leftovers. If rice repeatedly comes back, a useful next step might be a smaller default scoop with seconds available, followed by another round of observation. Taste and preference can also explain leftovers, so the app presents a question to investigate rather than declaring that a portion is too large.

How I built it

The frontend uses React, TypeScript, and Vite. A Python FastAPI backend handles uploads, analysis, review, and persistence in SQLite. The mask editor uses SVG polygons so food regions can be adjusted directly on the photograph.

For the local model, I trained a dense segmentation head on top of a frozen DINOv2 ViT-S/14 backbone using 1,732 FoodSeg103 training images. Its four target classes are rice, pasta, broccoli, and chicken; the source chicken label includes duck. I also explored different decoders, hard negatives, weak supervision from Nutrition5k ingredient labels, and partial backbone fine-tuning.

Challenges I ran into

Chicken was the hardest class. Changes that recovered more chicken could also produce more false positives or worse outlines. Food presence and mask overlap needed separate evaluation: finding chicken somewhere on a plate is different from outlining it accurately.

Another challenge was domain shift. Clean food photographs do not fully represent mixed leftovers under dining-hall lighting. I added other-food negatives and checked performance on Nutrition5k, but kept human review central to the product.

Accomplishments I'm proud of

The complete upload-to-save flow works: a real photo is analyzed, its food names are reviewed, and the confirmed record survives a refresh and reopens from Your scans.

A locally trained specialist achieved a higher measured mask IoU than Opus 5.5, Grok 4.7, and Sonnet 5.5 on the pilot described above. Original baselines are included, and unsupported scores remain unmeasured. The model comparison is separate from the broader food-name suggestions used by the OpenAI scanner.

What I learned

The useful unit of progress was a reviewed plate, not a confident model response. More training did not always improve the result, and a recall gain could hide a mask-quality regression. Keeping the examples, predictions, and evaluation records made those tradeoffs visible.

What's next for PlateBack

The next step is a fixed-camera prototype at a plate collection station: detect each arriving plate, avoid counting the same plate twice, run segmentation automatically, and flag uncertain records for staff review. Camera placement should capture plates rather than identify diners. I'd then run a small, permission-based dining-hall pilot and collect reviewed leftover photographs across repeated menus. That would provide a more realistic test set and a way to check whether portion changes actually reduce recurring leftovers. Next steps include more food categories, better small-piece masks, and optional scale measurements for any claims about food mass.