Inspiration
FoodLabel Guardian was inspired by a simple but important problem: food labels contain a large amount of information, but that information is not always easy for ordinary shoppers to interpret quickly and consistently.
Ingredient lists, additive codes, allergen information, serving sizes, nutrient quantities, processing classifications, and nutritional scores are often presented in formats designed primarily around regulatory requirements rather than everyday decision-making. The prototype therefore focuses on turning this dense information into something that can be searched, normalized, compared, checked against personal criteria, and explained in plain language.
The core idea was not to create another system that simply says whether a food is "good" or "bad." Instead, FoodLabel Guardian was designed around transparent comparison and user-defined criteria.
For example, different shoppers may care about completely different attributes:
avoiding ultra-processed foods; avoiding palm oil; limiting sugar; following a vegan diet; avoiding additives; identifying allergens; comparing protein, fiber, saturated fat, salt, sugar, or calorie levels.
The prototype reflects this philosophy through configurable rules rather than a single universal recommendation engine.
Another important inspiration was the possibility of combining fine-tuned AI models with agentic workflows. Rather than asking one language model to perform every task, the system separates the workflow into specialized components: product lookup, ingredient extraction, unit normalization, rule checking, and explanation.
This creates a more structured approach to food-label intelligence:
Scan → Extract → Normalize → Check → Compare → Explain
The resulting experience is intended to help users understand the information on a product without presenting the system as a medical diagnostic tool. The prototype explicitly maintains a "comparison only" constraint and reports zero medical claims as an MVP requirement.
What it does
FoodLabel Guardian is an interactive food-label intelligence and product-comparison platform.
At a high level, it takes product information, processes the ingredients and nutritional attributes through a multi-stage AI pipeline, evaluates the product against user-selected criteria, and enables side-by-side comparison with other products.
- Product scanning and import
The first stage allows a user to select a product from the prototype's sample product catalogue.
The simulation contains six representative products spanning categories such as beverages, snacks, dairy, confectionery, and plant-based beverages. These include examples such as Cherry Cola Zero, Oat & Honey Granola Bar, Plain Greek Yogurt, Sea Salt Potato Chips, Dark Chocolate 70%, and Barista Oat Drink.
The interface presents a normalized nutrition panel containing information such as:
serving size; calories; total fat; saturated fat; carbohydrates; dietary fiber; sugars; protein; salt; NOVA processing group; Nutri-Score; vegan status; organic status.
- Ingredient intelligence
FoodLabel Guardian applies a fine-tuned ingredient extraction model to the raw ingredient list.
The prototype demonstrates classification of ingredient entities into categories including:
Allergen Additive Sweetener Base ingredient
The interface visually highlights these different categories, allowing the user to see not merely the extracted ingredient text but also how the system has interpreted different components.
This is particularly important because ingredient lists can contain both familiar food ingredients and technical terms, additives, emulsifiers, preservatives, sweeteners, and allergen-related information.
- Personalized rule checking
Users can activate criteria that matter to them.
The current prototype implements five rule types:
Avoid NOVA 4 / ultra-processed foods No palm oil Low sugar — below 5 g/100 g Vegan only No additives / E-codes
The rule-checking agent evaluates the selected product whenever the active criteria change and reports whether the product passes or fails each selected rule.
This means the same product can produce different results depending on the user's priorities.
- Product comparison
Users can add scanned products to a comparison tray.
The prototype supports comparison of up to three products and normalizes the comparison to per-100 g/ml values.
The comparison includes:
calories; total fat; saturated fat; sugars; fiber; protein; salt; NOVA group; Nutri-Score; allergens; serving size.
The system highlights relative differences in the comparison table rather than relying exclusively on a single composite score.
- Explainable trade-offs
FoodLabel Guardian also includes an explanation model designed to turn numerical differences into a readable summary.
For example, the system can identify differences in sugar, processing group, and protein between two products and communicate those differences in natural language.
Importantly, the prototype's explanation explicitly avoids declaring one product universally "better." Instead, it frames the decision around trade-offs and user priorities.
- Shopping list
The application also includes a shopping-list workflow.
Products can be added to a personal list, and the interface displays product metadata such as brand, serving information, Nutri-Score, and the number of active rules that the product fails.
This provides a bridge between information discovery and practical shopping decisions.
- Evaluation dashboard
The prototype includes a simulated MVP metrics dashboard containing:
96.4% label extraction accuracy; 91.8% comparison correctness against dietitian-reviewed pairs; 62% repeat use within 14 days; 3.1% user correction rate.
These figures are presented in the source as simulated evaluation/MVP targets, rather than evidence of a completed production-scale clinical or consumer study.
How we built it
FoodLabel Guardian was structured as a hybrid fine-tuned AI + agentic pipeline.
The prototype identifies Open Food Facts and public-health sources as its underlying data foundation.
- Data foundation
The system uses the Open Food Facts schema as the foundation for product information.
The prototype represents a large product index and states that its demonstration is based on an Open Food Facts index of approximately 4.5 million products, while the interface displays a more specific indexed-product figure of 4,535,554.
The raw product representation is intentionally transformed into a smaller, comparison-ready structure. The prototype summarizes this as:
211 raw columns → 18 normalized fields
This normalization is important because product datasets can contain numerous heterogeneous fields, missing values, inconsistent units, and different representations of nutritional information.
- Fine-tuned ingredient extraction
The first AI component is an Ingredient NER model.
Its purpose is to transform unstructured ingredients_text into structured entities.
The architecture represented in the prototype separates:
Raw ingredient text → Fine-tuned NER → Typed ingredient entities
The extracted entities are then categorized into classes such as allergens, additives, sweeteners, and base ingredients.
- Fine-tuned explanation model
A second fine-tuned language model is responsible for converting structured comparison information into plain-language explanations.
This creates a division between:
Extraction intelligence
and
Explanation intelligence
rather than asking one model to simultaneously parse labels, perform numerical normalization, apply rules, and generate explanations.
- Agentic orchestration
The application organizes its workflow around five specialized agents:
Agent Primary responsibility Lookup Agent Matches a barcode/name to product information Extraction Model Parses ingredients into typed entities Unit Normalizer Converts nutritional values to comparable units Rule-Checking Agent Evaluates user-defined criteria Explainer Model Generates the trade-off explanation
The source explicitly describes this five-stage pipeline.
The architecture therefore resembles:
Product / Barcode │ ▼ ┌───────────────────┐ │ Lookup Agent │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Extraction Model │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Unit Normalizer │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Rule-Check Agent │ └─────────┬─────────┘ ▼ ┌───────────────────┐ │ Explainer Model │ └─────────┬─────────┘ ▼ Comparison + Explanation
- Deterministic rule engine
Rather than allowing the language model to make every decision, explicit rules are implemented programmatically.
For example, the prototype defines tests for:
NOVA ≥ 4; presence of palm oil; sugar ≥ 5 g/100 g; non-vegan status; presence of additives.
This hybrid design is valuable because deterministic rules make important parts of the system auditable and reproducible.
- Normalized comparison engine
The comparison layer operates on standardized nutritional quantities, primarily using a per-100 g/ml basis.
This avoids comparing products using incompatible serving sizes.
For example, a 20 g chocolate serving and a 330 ml beverage serving should not be compared directly without normalization. The prototype therefore explicitly labels its comparison table as all values per 100 g/ml.
- Interactive frontend
The prototype is implemented as a self-contained HTML application containing:
responsive CSS; product selection; simulated processing pipeline; nutrition rendering; ingredient entity highlighting; rule toggles; comparison tables; explanation panels; shopping-list management; evaluation metrics; live session indicator.
The interface is intentionally designed to make the underlying AI workflow visible rather than hiding it behind a single "Analyze" button.
Challenges we ran into
- Turning messy food-label data into structured information
One of the central challenges is the heterogeneity of food-product data.
The prototype specifically demonstrates a reduction from 211 raw columns to 18 comparison-ready fields, illustrating the amount of normalization required before products can be meaningfully compared.
The challenge is not simply collecting more data. It is establishing a reliable canonical representation of each product.
- Understanding ingredient lists
Ingredient lists are semi-structured natural language.
The same concept may be represented in different ways, while individual ingredients can have multiple roles. A useful system therefore needs to identify entities while preserving their original context.
This motivated the use of a dedicated fine-tuned NER component rather than relying solely on generic text generation.
- Unit and serving-size normalization
Nutritional labels can express information using different serving sizes.
A comparison engine therefore needs a consistent reference basis. FoodLabel Guardian addresses this through its Unit Normalizer and its per-100 g/ml comparison methodology.
- Avoiding overconfident health claims
Food information can easily drift into medical or diagnostic territory.
The project deliberately establishes a boundary: FoodLabel Guardian compares product information; it does not diagnose users or provide medical guidance.
The prototype explicitly reports zero medical claims as part of its MVP constraint and reinforces this in its explanation layer.
- Balancing automation with user control
Another challenge was deciding how much the system should decide automatically.
Instead of hiding the decision logic inside a black-box recommendation score, FoodLabel Guardian exposes user-configurable rules. This makes it possible for users to determine which criteria matter to them.
- Explaining trade-offs without oversimplifying them
Nutrition comparisons are multidimensional.
A product can have less sugar but more fat, more protein but a different processing classification, or a favorable nutritional attribute alongside an allergen.
The explanation component therefore needs to communicate trade-offs rather than reduce everything to a single simplistic answer. The prototype explicitly uses this principle in its comparison explanation.
- Building a convincing MVP before full-scale deployment
The current implementation is a simulation rather than a complete production system. The six products are representative sample records, while the metrics dashboard is explicitly described as a simulated evaluation dashboard and MVP targets.
Consequently, moving from prototype to production will require substantially larger validation datasets, real barcode/image ingestion, systematic model evaluation, and real-world user testing.
Accomplishments that we're proud of
- We created a complete end-to-end concept
FoodLabel Guardian is more than an ingredient parser.
It demonstrates a complete workflow:
Product discovery → extraction → normalization → personalized rules → comparison → explanation → shopping workflow
That makes the project a concrete demonstration of how fine-tuned models and agentic components can cooperate in a consumer application.
- We made the AI pipeline visible
Rather than hiding the processing stages, the interface explicitly displays the five stages of the pipeline.
This provides users with a conceptual understanding of what happens to their product data between scanning and the final comparison.
- We combined AI with deterministic logic
The system does not depend entirely on generative AI.
The fine-tuned models handle natural-language tasks, while deterministic programmatic rules handle criteria that can be expressed explicitly.
This creates a more controllable architecture for a domain where traceability matters.
- We made comparison the central interaction
The application does not force users into a universal product ranking.
Instead, it lets them place products side by side and inspect differences across multiple nutritional and ingredient dimensions.
- We built explainability directly into the workflow
The "Explain the trade-off" functionality provides an additional interpretation layer after the numerical comparison.
This is an important accomplishment because it connects structured data with human-readable reasoning without making the explanation itself the only source of truth.
- We incorporated personalized criteria
The configurable rule system means FoodLabel Guardian can support different shopping objectives rather than assuming that every user has the same priorities.
- We established measurable MVP targets
The prototype does not stop at interface design. It defines measurable targets covering extraction accuracy, comparison correctness, repeat usage, and correction rate.
These provide a foundation for transitioning from a demonstration to empirical evaluation.
What we learned
- The most useful AI systems are often hybrid
FoodLabel Guardian reinforced the value of combining:
fine-tuned ML + deterministic algorithms + agent orchestration + structured data
A language model is useful for interpreting unstructured language, but explicit rules are often better for clearly defined conditions.
- Data normalization is as important as model intelligence
A sophisticated model cannot compensate for inconsistent inputs.
The 211-to-18 normalization concept demonstrates that building a reliable canonical data layer is fundamental to downstream comparison quality.
- Explainability should be part of the architecture
Explanation should not be treated as an afterthought.
The system works better conceptually when the architecture preserves the information needed to explain why two products differ.
- Personalization should be explicit
Different people legitimately care about different criteria.
Making rules configurable provides a transparent mechanism for personalization without requiring the model to infer a user's priorities.
- Health-adjacent applications require careful boundaries
The project demonstrates the importance of distinguishing food-information interpretation from medical advice.
The prototype's comparison-only framing provides a useful foundation for maintaining that distinction.
- User corrections can become valuable training signals
The prototype tracks a correction rate of extracted fields as one of its proposed metrics.
In a production implementation, these corrections could become an important feedback mechanism for improving extraction models, identifying systematic errors, and prioritizing additional training examples.
- Trust comes from showing the reasoning pathway
Users are more likely to understand an AI-assisted food-information system when they can see the underlying ingredients, normalized numbers, active rules, comparison results, and explanation separately.
That separation is one of the strongest architectural ideas demonstrated by the prototype.
What's next for FoodLabel Guardian
The next stage is to evolve FoodLabel Guardian from a simulation/MVP demonstration into a production-ready, multimodal food-label intelligence platform.
- Real barcode and camera scanning
The product picker should be replaced or supplemented with real-world input:
Camera → Barcode/OCR → Product Lookup → Label Analysis
A user should be able to point a smartphone camera at a product barcode or ingredient label and receive an analysis without manually selecting a sample product.
- Multimodal label understanding
The next-generation system could combine:
OCR; barcode recognition; computer vision; ingredient NER; nutrition-table extraction; entity normalization; product-database retrieval.
This would allow the system to work even when a product is not already perfectly represented in the underlying database.
- Larger fine-tuning datasets
The ingredient extraction model should be expanded using carefully annotated food-label data covering:
ingredients; allergens; additives; sweeteners; preservatives; emulsifiers; oils; flavorings; processing terms; ambiguous ingredient phrases.
The explanation model could similarly be trained on high-quality, expert-reviewed comparison examples.
- Evidence-backed explanations
Future explanations should link individual claims to their underlying evidence:
Ingredient → extracted entity → rule → nutritional value → comparison → explanation
This would make the system more auditable and allow users to inspect why a particular flag was generated.
- Regional and international food-label support
A production version could support different regulatory and labeling conventions across countries.
This is particularly relevant for international products, where ingredient terminology, nutrition-label formats, allergen declarations, additives, and front-of-pack systems can differ.
- Personalized profiles
Users could optionally configure persistent preferences such as:
dietary pattern; allergens to watch; ingredient exclusions; nutritional thresholds; shopping objectives.
The system could then apply those preferences consistently while keeping the underlying rules visible and editable.
- Smarter agentic workflows
The five-agent architecture could eventually evolve into a larger orchestration layer capable of:
retrieving missing product information; resolving conflicting product records; checking multiple authoritative sources; detecting missing label fields; requesting additional information from the user; validating extracted values; generating evidence-backed comparisons; learning from user corrections.
The important principle would remain controlled agentic behavior rather than unrestricted autonomous recommendations.
- Personal shopping intelligence
The shopping-list functionality could become a larger workflow.
For example:
Scan products → Apply personal rules → Compare alternatives → Save qualifying products → Build shopping list
This could turn FoodLabel Guardian from a label reader into a broader food-shopping intelligence assistant.
- Production-grade evaluation
The simulated metrics should eventually be replaced with independently measured results.
Evaluation could include:
ingredient entity precision/recall/F1; nutrition-field extraction accuracy; OCR accuracy; barcode lookup accuracy; rule-evaluation accuracy; comparison correctness; explanation factuality; hallucination rate; user correction rate; latency; repeat usage; usability; robustness across brands and label formats.
The current prototype already provides a useful starting framework through its extraction, comparison, user-correction, and repeat-use metrics.
- Responsible deployment
Finally, the production system should preserve the project's core philosophy:
Read the label. Understand the evidence. Compare the trade-offs. Let the user decide.
FoodLabel Guardian's strongest future direction is therefore not simply becoming a more powerful recommendation engine. It is becoming a transparent, evidence-oriented, personalized food-label intelligence platform where AI handles the difficult extraction and interpretation work while users retain visibility and control over the criteria being applied.
Overall, the prototype establishes a strong foundation: Open Food Facts provides the product-data foundation, fine-tuned models handle specialized language understanding, agents coordinate the workflow, deterministic rules provide controllability, and explainable comparisons turn complex label information into something a shopper can actually use.
Log in or sign up for Devpost to join the conversation.