Retail Intelligence Model

A product matching model that identifies whether product listings from different retailers refer to the same real-world product, even when their names and descriptions are completely different.

Comment

Retail Intelligence Model

An AI-powered product matching system that identifies whether product listings from different retailers refer to the same real-world product, even when their names and descriptions are completely different.

Dataset used : https://www.kaggle.com/datasets/saanvibeledenaga/retail-intelligence-platform

Inspiration

E-commerce comparison shopping seems simple on the surface, but the moment you look at raw retailer data, the problem becomes clear: the same physical product can be described completely differently across platforms.

For example:

Retailer A: Sony 40in LCD
Retailer B: Sony BRAVIA 40-Inch LCD HDTV

There is often no shared identifier, universal barcode, or guarantee that the same brand and model will be phrased the same way.

This inspired us to apply entity resolution, a well-established problem in data engineering, to real-world retail catalogs.

If price-comparison tools and marketplaces could reliably determine that two listings represent the same product without relying on a shared identifier, it would enable:

  • Accurate cross-retailer price comparison
  • Competitive pricing analysis
  • Cleaner marketplace catalogs
  • Product deduplication
  • Retail intelligence at scale

What It Does

The Retail Intelligence Platform takes product listings from two different retailers and automatically predicts whether any given pair refers to the same real-world product.

The system:

  • Narrows down candidate pairs from over 1 million possible comparisons to a small, relevant shortlist
  • Uses a fine-tuned Sentence-BERT model to understand semantic similarity beyond exact word matching
  • Combines multiple similarity signals including semantic similarity, word overlap, price consistency, and model-number matching
  • Uses a LightGBM classifier to make the final match/no-match decision
  • Outputs precision, recall, F1-score, and accuracy
  • Provides visual analysis of how and why the model makes its decisions

How We Built It

The system follows a four-stage entity resolution pipeline:

1. Blocking

Comparing every product against every other product is computationally expensive.

For our dataset:

1,081 × 1,092 ≈ 1.18 million possible pairs

Product names are encoded using a fine-tuned Sentence-BERT model.

Cosine similarity is calculated between product listings, and only the top 10 most similar candidates per product are retained.

~1.18 million possible pairs
              ↓
       Semantic Similarity
              ↓
       Top-10 Candidates
              ↓
       Manageable Search Space

This dramatically reduces the number of pairs that need to be processed by the downstream classifier.

Important: Blocking creates a hard ceiling on overall recall. If a true match is removed during blocking, the classifier cannot recover it later. Therefore, blocking recall is explicitly evaluated against the ground truth.

2. Fine-Tuning Sentence-BERT

Instead of using Sentence-BERT out of the box, we fine-tuned:

all-MiniLM-L6-v2

on labeled match / non-match product pairs.

This teaches the model domain-specific notions of product similarity.

For example:

Sony BRAVIA 40-Inch LCD HDTV
Sony 40in LCD

should be recognized as highly similar even though the wording is different.

The fine-tuned model learns that signals such as brand names and model numbers are often more important than generic descriptive language.

3. Feature Engineering

For every candidate pair, multiple complementary features are calculated.

Semantic Features

  • Fine-tuned Sentence-BERT similarity

Lexical Features

  • Word overlap
  • Character-level similarity
  • Description overlap

Product-Specific Features

  • Exact model-number overlap
  • Brand information

Numerical Features

  • Price difference
  • Price consistency

Combining these signals allows the model to evaluate product identity from multiple perspectives rather than relying on a single similarity metric.

4. Classification

The engineered features are passed into a LightGBM gradient-boosted classifier.

The classifier produces the final:

MATCH
   or
NO MATCH

prediction.

Because genuine product matches make up only a tiny fraction of all candidate pairs, the dataset is highly imbalanced.

A weighted loss is therefore used to prevent the classifier from simply predicting NO MATCH for most examples.

Evaluation

The model is evaluated using:

  • Precision
  • Recall
  • F1-score
  • Accuracy

The primary evaluation metric is the F1-score, which balances precision and recall.

F1 Score

              2 × Precision × Recall
F1 = ─────────────────────────────────────
           Precision + Recall

F1 is particularly useful for this task because true product matches are relatively rare, making accuracy alone potentially misleading.

Challenges We Ran Into

Encoding Issues in Raw Data

The source CSV files were not UTF-8 encoded.

Special characters commonly found in product names caused decoding errors until latin-1 encoding was applied consistently.

Blocking Recall as a Hard Ceiling

Blocking improves scalability but introduces a fundamental limitation.

If a true match is missed during blocking:

Blocking
   ↓
True Match Removed 
   ↓
Classifier Never Sees It

The downstream classifier cannot recover it.

This made measuring blocking recall against ground truth an essential part of the evaluation process.

Class Imbalance

True matches represented only a tiny fraction of candidate pairs.

An unweighted classifier initially learned to favor:

NO MATCH

This resulted in misleadingly high accuracy while performing poorly on the actual matching task.

We addressed this using weighted classification.

Infrastructure Quirks

Fine-tuning larger models on a multi-GPU session triggered automatic multi-GPU wrapping, which caused training failures.

The training environment was explicitly restricted to a single GPU to resolve the issue.

Notebook State Management

The pipeline contains several interdependent stages.

During development, partial notebook reruns occasionally left the notebook in an inconsistent state.

This reinforced the importance of:

Restart → Run All → Validate

before treating results as final.

Accomplishments We're Proud Of

  • Built a complete entity resolution pipeline from raw retail data to final predictions
  • Reduced over 1 million potential comparisons to a manageable candidate set using blocking
  • Fine-tuned a Sentence-BERT model specifically for retail product matching
  • Combined semantic, lexical, numerical, and product-specific signals
  • Built a LightGBM-based classification system
  • Addressed severe class imbalance using weighted classification
  • Achieved strong precision, recall, F1, and accuracy on unseen test data
  • Demonstrated through visual analysis that fine-tuning improved matching performance
  • Designed the system using a scalable blocking → feature engineering → classification architecture

What We Learned

1. Blocking Is Essential

Naively comparing every product against every other product does not scale.

Blocking is therefore a fundamental component of entity resolution rather than simply an optimization.

2. Off-the-Shelf Embeddings Aren't Enough

Generic embedding models are not automatically domain-aware.

Fine-tuning on labeled examples can significantly improve the model's understanding of what constitutes a meaningful product match.

3. No Single Similarity Signal Is Enough

Semantic similarity alone can produce false matches.

Combining:

Semantic Similarity
        +
Word Similarity
        +
Character Similarity
        +
Model Number Matching
        +
Price Consistency
        +
Description Similarity

produces a more reliable representation of product identity.

4. Accuracy Can Be Misleading

When true matches are rare, a model can achieve high accuracy while failing to identify actual matches.

Precision, recall, and F1-score therefore provide a more meaningful evaluation.

5. Pipeline Management Matters

Model quality is only one part of a reliable ML system.

Maintaining consistent notebook state and validating the complete pipeline is equally important when working with multi-stage workflows.

Tech Stack

Component Technology
Language Python
Embedding Model Sentence-BERT
Base Model all-MiniLM-L6-v2
Fine-Tuning Sentence Transformers
Classifier LightGBM
Similarity Cosine Similarity
Data Processing Pandas, NumPy
Evaluation Precision, Recall, F1, Accuracy
Development Kaggle Notebook
Hardware GPU

Pipeline

             ┌──────────────────────┐
             │   Retailer Catalog A │
             └──────────┬───────────┘
                        │
                        │
             ┌──────────▼───────────┐
             │   Retailer Catalog B │
             └──────────┬───────────┘
                        │
                        ▼
              ┌───────────────────┐
              │ Data Preprocessing│
              └─────────┬─────────┘
                        │
                        ▼
              ┌───────────────────┐
              │ Fine-Tuned SBERT  │
              └─────────┬─────────┘
                        │
                        ▼
              ┌───────────────────┐
              │     Blocking      │
              │     Top-K = 10    │
              └─────────┬─────────┘
                        │
                        ▼
              ┌───────────────────┐
              │ Feature Engineering│
              ├───────────────────┤
              │ Semantic Similarity│
              │ Word Overlap       │
              │ Character Similarity│
              │ Description Overlap│
              │ Price Difference   │
              │ Model Number Match │
              └─────────┬─────────┘
                        │
                        ▼
              ┌───────────────────┐
              │     LightGBM      │
              │    Classifier     │
              └─────────┬─────────┘
                        │
                        ▼
                 ┌──────────────┐
                 │ MATCH /      │
                 │ NO MATCH     │
                 └──────────────┘
                        │
                        ▼
              ┌───────────────────┐
              │ Evaluation &      │
              │ Visualization     │
              └───────────────────┘

What's Next?

We plan to extend the Retail Intelligence Platform by:

  • Scaling the pipeline to larger, real-world product catalogs
  • Incorporating additional attributes such as manufacturer and category
  • Adding product specifications such as dimensions, color, storage, and screen size
  • Exploring ensembles of multiple embedding models and classifiers
  • Building a lightweight API for integration with external applications
  • Developing an interactive interface for inspecting matches and confidence scores
  • Integrating the system into a real-world price comparison or competitive pricing platform

Goal

The ultimate goal of the Retail Intelligence Platform is to make product identity resolution reliable and scalable, even when retailers provide inconsistent or differently formatted product information.

By combining domain-adapted language models with traditional entity resolution techniques, the platform provides a foundation for smarter:

Price Comparison • Competitive Pricing • Marketplace Management • Retail Intelligence

Built With

  • data-science
  • deep-learning
  • ecommerce
  • entity-resolution
  • lightgbm
  • machine-learning
  • natural-language-processing
  • price-comparison
  • product-matching
  • product-matching-system
  • retail-intelligence
  • sbert
  • semantic-search
  • sentence-transformers
Share this project:

Updates

Submission history