Inspiration
E-commerce platforms process millions of product listings daily from thousands of third-party vendors. A major catalog challenge is entity resolution—identifying whether two differently named or formatted items from different sources represent the exact same physical product. Inspired by the scale of modern catalog management, we built NovaMatch to automate and optimize product matching using state-of-the-art machine learning.
What it does
NovaMatch is an intelligent entity resolution and product matching pipeline that:
- Cleans and Normalizes raw product attributes (titles, descriptions, brand names, and specifications).
- Performs Fast Blocking to group candidate matches and reduce computational overhead.
- Scores Semantic Similarity using transformer-based models to confirm or reject item matches with high precision.
- Provides an Interactive Dashboard where catalog managers can review flagged matches, approve resolutions, and visualize confidence metrics.
How we built it
- Backend & ML Pipeline: Built using Python, leveraging PyTorch and Hugging Face Transformers for deep semantic embeddings, alongside Scikit-learn for feature extraction and blocking logic.
- API Layer: Wrapped the inference engine in a robust FastAPI backend to handle real-time matching requests.
- Frontend: Developed an interactive interface using Streamlit for seamless catalog visualization and review.
- Infrastructure: Containerized using Docker with PostgreSQL for storing match history and catalog logs.
Challenges we ran into
- Messy Data: Real-world product titles contain typos, missing attributes, and inconsistent ordering (e.g., "Apple iPhone 15 Pro 128GB Black" vs. "iPhone 15 Pro - 128 GB"). We designed robust text preprocessing and normalization steps to overcome this.
- Scalability: Comparing every product against every other product ($O(N^2)$ complexity) is too slow at scale. We solved this by implementing efficient blocking algorithms to narrow down candidate pairs before running deep learning inference.
Accomplishments that we're proud of
- Achieving high precision and recall on duplicate product detection without crippling latency.
- Delivering a complete end-to-end prototype from raw data ingestion to an intuitive UI dashboard.
What we learned
- The importance of balancing computational speed with matching accuracy in production ML systems.
- How effective blocking strategies can make or break an entity resolution pipeline.
Built With
- docker
- fastapi
- python
- pytorch
- scikit-learn
- streamlit
- transformers
Log in or sign up for Devpost to join the conversation.