Sivle Copilot: Conversational Search & Recommendations

Project Description

1. How the Solution Addresses the Problem Statement

Our solution, Shopping Copilot, is an intelligent, high-performance conversational commerce agent designed to bridge the gap between ambiguous user queries and a 50,000-item e-commerce catalog. Built upon the four competition pillars:

Core Architecture (Intent Routing & Hybrid Retrieval): Employs a dual-track intent router splitting queries into high-precision BUYING (hard constraint filtering) and exploratory BROWSING (broad semantic discovery). Retrieval is powered by an in-memory SQLite FTS5 multi-field BM25 engine with tuned column weights, followed by multi-signal semantic re-ranking (category alignment, constraint coverage, negation suppression, and Bayesian rating/popularity priors).
Dialog Strategy (Multi-Turn Dynamic State Machine): Tracks conversational slots across turns, handling progressive constraint accumulation and zero-latency Intent Overrides (automatic slot erasure of invalidated preferences when a user changes their mind). Proactive clarification uses a Universal Probing strategy (ask_attribute = "other") to guide user convergence rapidly.
Self-Evolution (Context Programming & Personalization): Dynamically refines session-level constraint representations and aligns retrieved candidates with user profile preference tags.
Evaluation & Commercial Efficiency: Achieves a verified 98.0% Hit Rate@10, 0.7285 MRR, and collapses Mean Turns to Conversion (MTTC) to 2.290 turns on the 200 public evaluation sessions, yielding a 0.8828 Recommended Technical Score with a lightweight token footprint (~159 tokens/session).

2. Development Tools Used

IDEs & Editors: Visual Studio Code, Antigravity IDE, Cursor
Terminal & Environments: macOS Zsh, Python 3.12 virtual environments, Node.js 18+ runtime
Package & Task Managers: uv (ultra-fast Python package resolver), npm, Vite build tooling

3. APIs Used

FastAPI REST API: High-throughput asynchronous backend service providing endpoints for session turn processing (/api/sessions/turn), state inspection, and catalog exploration.
OpenAI-Compatible Inference API (NUS SoC Class Llama 3.1 8B Endpoint / LangChain): Cloud-hosted LLM endpoint for deep semantic slot extraction on complex conversational queries, intent override reasoning, and real-world token telemetry, coupled with a deterministic offline fallback.

4. Libraries and Frameworks Used

Backend & NLP: FastAPI, Uvicorn, LangChain, LangChain-OpenAI, LangGraph, Pydantic, Python-Dotenv, SQLite3 FTS5 module.
ORM & Database: Prisma Client Python, SQLite (in-memory and relational persistence).
Frontend: React 18, Vite, Tailwind CSS, Lucide React, Axios.

5. Datasets and Assets Used

Amazon Reviews 2023 Dataset: Frozen catalog of 50,000 products in the Clothing_Shoes_and_Jewelry category (data/catalog.jsonl).
Official Public Benchmark Dataset: 200 labeled public multi-turn evaluation sessions (data/public_set.jsonl).
Synthetic Evaluation Benchmark Suite: 5 generated randomized 200-sample test sets across standard, buying-heavy, browsing-heavy, and override-heavy scenario mixtures (test-data/).

About the project

Inspiration

Traditional e-commerce search engines are rigid keyword filters. When shoppers browse online, their desires evolve dynamically: they start with broad categories, add nuanced constraints, and frequently change their minds mid-conversation. Heavy vector databases and multi-billion-parameter LLM chains often introduce high latency, unpredictable hallucinations, and steep API costs that make conversational shopping impractical in production.

We were inspired to build Shopping Copilot — a blazing-fast, state-aware conversational agent that unites deterministic state machine precision with high-performance in-memory text retrieval, delivering sub-millisecond candidate ranking and natural conversational convergence.

What It Does

Shopping Copilot is a dual-track conversational e-commerce engine that guides users from open-ended browsing to high-intent product conversion within an average of 2.29 conversational turns:

Dual Execution Environments:
1. Headless Evaluator Engine: Evaluates hundreds of multi-turn sessions in seconds with zero HTTP overhead.
2. Interactive Full-Stack Web App: A dual-pane playground featuring an interactive shopping copilot alongside a real-time State Inspector displaying active slots, negative filter erasures, token telemetry, and ranked product cards.
Sub-Millisecond Candidate Retrieval: Indexes 50,000 catalog items in an in-memory SQLite FTS5 table with field-weighted BM25 ranking.
Dynamic Intent Override & Slot Erasure: Detects when shoppers negate earlier preferences (e.g., "Actually, ignore my earlier preference. What I need is...") and instantly purges outdated constraints.
Universal Probing: Asks dynamic, non-intrusive clarification questions (ask_attribute = "other") to prompt shoppers for their highest-priority latent constraints.

How We Built It

┌─────────────────────────────────────────────────────────────────────────┐
│                       User Conversational Query                         │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Stage 1: Intent Routing & State Machine Tracker                         │
│ • Dual-track classification (BUYING vs. BROWSING)                       │
│ • Regex slot extraction & dynamic override slot erasure                 │
│ • Optional selective LLM fallback (LangChain)                           │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Stage 2: Universal Probing Strategy                                     │
│ • ask_attribute = "other" for rapid constraint convergence (MTTC: 2.29) │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Stage 3: In-Memory SQLite FTS5 Multi-Field Retrieval                    │
│ • Weighted BM25: Title (12.0), Category (12.0), Features (9.0), etc.   │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ Stage 4: Multi-Signal Semantic Re-Ranking & Bayesian Prior              │
│ • Category ratio + constraint bonuses + rating & popularity priors      │
└────────────────────────────────────┬────────────────────────────────────┘
                                     │
                                     ▼
                      Top-10 Recommended Products

1. In-Memory Multi-Field SQLite FTS5 Retrieval

The 50,000-item catalog is loaded into an in-memory SQLite FTS5 virtual table with unicode61 remove_diacritics 2 tokenization. We apply tuned column-specific BM25 weights:

Title: 12.0 (Primary product identifier)
Category: 12.0 (Enforces domain taxonomy)
Features: 9.0 (Captures material, fit, and technical attributes)
Details: 2.0 (Catches metadata specifications)
Store & Description: 1.0 (Broad contextual matching)

2. Multi-Signal Semantic Scoring Function

Retrieved candidates are re-ranked using our composite scoring formula:

$$S(p, q) = -\text{BM25}{\text{raw}} + \alpha{\text{cat}} \cdot \frac{|C_q \cap C_p|}{|C_q|} + \sum_{c \in K_q} \beta_c + \gamma \cdot (R_p - 3.0) + \delta \cdot \ln(1 + N_p) - \Omega_{\text{neg}}$$

Where:

( -\text{BM25}_{\text{raw}} ) is the negated SQLite BM25 value (converting lower-is-better to higher-is-better).
( \alpha_{\text{cat}} ) is the category alignment bonus (( 12.0 ) for BUYING, ( 9.0 ) for BROWSING).
( \beta_c ) is the constraint match weight (( +4.5 ) per corpus match, ( +3.0 ) for title match, ( +5.0 ) for complete constraint satisfaction).
( R_p ) and ( N_p ) represent the product's average star rating and review count (Bayesian popularity prior).
( \Omega_{\text{neg}} ) is the penalty (( -8.0 )) applied to products containing negated attributes after an intent override.

3. Evaluation & Optimization

The system was optimized against the official competition evaluation metric:

$$\text{Efficiency} = \max\left(0, \frac{11 - \text{MTTC}}{10}\right)$$

$$\text{TechnicalScore} = 0.50 \cdot \text{HitRate@10} + 0.30 \cdot \text{MRR} + 0.20 \cdot \text{Efficiency}$$

Verified Benchmark Results

On the official 200 Public Test Sessions (data/public_set.jsonl), Shopping Copilot achieved:

Metric	Target / Starter Baseline	Our Verified Result	Performance Delta
Hit Rate@10	`0.5000`	`0.9800` (98.0%)	+48.0%
MRR (Mean Reciprocal Rank)	`0.3000`	`0.7285`	+0.4285
MTTC (Mean Turns to Conversion)	`5.0000`	`2.2900` turns	-2.710 turns faster
Efficiency Score	—	`0.8710`	Optimal
Recommended Technical Score	—	`0.8828`	Top-Tier
Total Reported Token Usage	—	`31,825` tokens	~159 tokens / session

Scenario Performance Breakdown:

Boundary Cases (10 sessions): 100.0% Hit Rate@10 | 0.9500 MRR | 2.60 MTTC
Browsing Scenarios (80 sessions): 100.0% Hit Rate@10 | 0.6750 MRR | 1.96 MTTC
Buying Scenarios (80 sessions): 96.25% Hit Rate@10 | 0.7085 MRR | 1.96 MTTC
Intent Override Scenarios (30 sessions): 96.67% Hit Rate@10 | 0.8509 MRR | 3.93 MTTC

Challenges We Faced

Handling Abrupt Intent Overrides: When customers say "Actually, ignore my earlier color preference", standard vector accumulation pollutes the search context with the old keyword. We solved this by building an explicit Slot Erasure Engine that purges invalidated constraints and applies negative score penalties ( \Omega_{\text{neg}} ) to previously suggested items.
Balancing Latency with Conversational Quality: Heavy dense embeddings and cross-encoders introduce unacceptable latency and require dedicated GPU clusters. By optimizing an in-memory SQLite FTS5 index with custom BM25 column weights, we achieved sub-millisecond retrieval without sacrificing precision.
Proactive Clarification Dilemma: Asking narrow single-attribute questions (e.g. asking only for "color" when the user cares about "waterproofing") wastes turns. Designing the Universal Probing Policy (ask_attribute = "other") allowed simulated customers to provide their highest-priority constraints immediately, slashing MTTC to 2.29 turns.

What We Learned

State Representation > Model Size: High-precision conversational search depends heavily on robust state management and slot tracking rather than simply invoking larger foundation models.
Domain-Specific Column Weighting: Calibrating BM25 weights across distinct metadata fields (Title: 12.0 vs. Description: 1.0) dramatically suppresses false-positive noise in product catalogs.
Proactive Conversational Acceleration: Structuring clarification prompts to allow users to disclose multiple free-form constraints dramatically reduces conversational cognitive load.

What's Next for Shopping Copilot

Two-Stage Quantized Cross-Encoder Reranking: Integrating a lightweight local ONNX cross-encoder (ms-marco-MiniLM-L-6-v2) on the top-50 FTS5 candidates to capture zero-overlap synonyms in <15ms.
Long-Term Persona Distillation: Incorporating historical customer purchase graph embeddings to personalize initial turn candidate selection.
Multi-Modal Style Visual Search: Enabling visual similarity search over product image embeddings to support visual discovery.