Label Guard

Label Guard judges whether a phone photo of a food label is readable, then retakes, warns or asks a human; it reports how often it abstains and how often it still misses silently.

Comment

Inspiration

A person with a food allergy photographs the ingredient list on the back of a package. Phone photos are often blurred, glared or tilted, and a tool that reads a bad photo can still say "no milk found": a silent miss. The question we answer is not only "what does the label say?" but "is this photo good enough to say an allergen is absent?" If not, the right output is an action, not a guess.

What it does

Agentic Vision path: OpenCV 5 output changes the next action. OpenCV 5 first judges photo quality (perspective rectification attempted on every photo, transform applied on 3 of 104 public photos since most are already-cropped ingredient shots; then blur, glare, text regions). That result decides the next step: a retake request (the reader is not called), or read with the OpenCV 5 DNN text reader and then allergy warning, human check when the read is uncertain, or no warning. Every answer ends with "final check by a person, on the original label." It does not give medical advice and never says a product is safe to eat. The "agent" is a deterministic policy, not a language model; no generative model is called and no model was trained.

How we built it

  • OpenCV 5 (opencv-python-headless==5.0.0.93): Canny + findContours + approxPolyDP + getPerspectiveTransform/warpPerspective, Laplacian-variance blur score, morphology-based glare blobs, gradient/Otsu text regions, and cv2.dnn (DB text detection + PP-OCRv3 recognition, public pre-trained ONNX weights).
  • AWS: Lambda (python3.12, 2048 MB) behind a public Function URL, two Lambda layers, CloudWatch Logs, a logs-only IAM role, reserved concurrency 5. It fits the always-free tier.
  • Clean Architecture (domain / application / adapters / infrastructure); the reader is swappable behind a port. 81 unit tests (incl. 7 SAM-template checks).

Measured results (104 public Open Food Facts photos, deployed reader)

Silent miss = positive photo answered “no warning” (the most serious error). Recall = warned positives / positives; safe recall = (warned + retake or human check) / positives.

  • Photo-level warning recall 0.87, precision 0.80, safe recall (warned or abstained) 0.91, 5 silent misses out of 55 positive photos.
  • Held-out check (thresholds picked on 52 photos, scored on the other 52): recall 0.86 (24/28), 2 silent photos; per group x photo recall 0.75 (30/40) with 8 silent misses at the deployed confidence 0.6 (6 at 0.8, not deployed).
  • Per ingredient group the recall is lower (0.74 overall; fish 0.33, egg 0.56).
  • 1,248 synthetic retake pairs: verdict change rate 0.38; abstention 0.38 before retake, 0.10 after; useful abstention only 0.26.
  • Live endpoint (30 requests): 30/30 HTTP 200, round-trip p50 3.36 s; a blurred photo returns a retake request with a how-to-reshoot tip (hold steady, 10-15 cm, tap to focus) in 0.2-0.4 s (0.23 s in the 30-request test) because the reader is skipped.

Challenges

Reporting honestly: our abstention is too cautious (0.26 of abstentions were needed) and too weak for glare (strong glare is caught in only 11% of photos). Turning the quality gate off changes recall by one warning on already-cropped photos, so we only claim its effect for blur and low resolution.

Limitations and responsible use

  • The 1,248 retake pairs are synthetic degradations. The only real photos are 42 pairs of different Open Food Facts uploads of the same label (not one person retaking; mostly French): verdict agreement 0.76 (32/42, 95% CI 0.61-0.87), one side abstained 0.14 (6/42, always the worse photo), abstention useful 8/8, accuracy when answering 0.78 (59/76); with the gate off the same 76 photos score 0.78 too, and over all 84 photos 0.70 (the gate hands 8 would-be wrong answers to a person). Silent misses: 3 of 56 positive photos were answered "no warning" (one passed the blur gate at blur 24 and was misread). Same-person phone retakes: planned, not measured.
  • Thresholds were tuned on the same 104 photos (a 52/52 split check held recall at 0.86, but 28 held-out positives is small). Ground truth is crowd-sourced, so precision is likely understated. No user study.
  • Not medical advice. Photos are processed in memory and discarded; logs hold only verdict kind, latency and group count. Photos are CC BY-SA from Open Food Facts; the repository holds no images.
  • AI use: the code was written with an AI coding assistant; we do not present that as the agentic workflow.
  • Similar tools (Fig, Yuka, Seeing AI, Be My AI, general VLM apps), per each tool's public page only, not hands-on and not benchmarked: they scan barcodes against a database, read text aloud, or describe a photo; general photo Q&A apps (ChatGPT, Gemini) answer free-form questions about an uploaded photo. Their public pages do not describe a photo-quality check or a measured abstention rate. We do not claim better reading accuracy; the claim is limited to measured abstention.

What we learned

A refusal to answer is a feature only if you measure it: we publish how often the abstention was right (0.26) and how often the system still missed (83 of 1,248 degraded photos).

What's next

Same-person phone retake pairs, Korean real pairs, glare-specific gating, ingredient synonym coverage for French/Spanish/Korean fish and egg terms.

Built With

Share this project:

Updates

Submission history