Stateful BM25 E-Commerce Search & Clarification Agent

1. How the Solution Addresses the Problem Statement

Real-world e-commerce dialogue requires navigating incomplete customer intents across multiple conversational turns. Standard search engines struggle when users provide sparse constraints up front, and rigid filters fail to capture evolving context.

Our solution directly addresses this challenge by engineering a stateful, zero-external-dependency conversational retrieval pipeline built on SQLite’s FTS5 full-text search engine. Instead of relying on expensive or unpredictable large language model (LLM) calls, our agent delivers deep commercial efficiency, deterministic execution, and low-latency performance through four core mechanisms:

• State Accumulation Across Turns: The agent maintains a persistent multi-turn conversational context (state["terms"]). Rather than treating turn $N$ as an isolated query, tokens from preceding customer messages accumulate across the session, allowing information provided in turn 1 to continuously guide retrieval through turn 8.
• Adaptive Attribute Clarification: The agent systematically tracks previously asked product attributes and sequentially prompts the user for unasked metadata (use_case, feature, material, color, style, size, budget) via ask_attribute. This systematically unlocks precise constraints from simulated customers who would otherwise remain vague.
• Deterministic Price Constraint Extraction: A regex parser captures explicit numeric price constraints (e.g., "under $80") directly from incoming messages and filters candidate items against an in-memory price lookup table without dropping items with missing/unknown price data.
• Custom Field-Weighted BM25 Ranking: Full-text matching ranks products dynamically across seven catalog attributes (title, categories, features, details, store, description). We tuned SQLite FTS5 column weights to prioritize title matches (weight 6.0) and feature highlights (weight 2.5), optimizing key term alignment.

Through empirical iteration, our streamlined rule-based architecture improved the baseline technical evaluation score from 0.107 to 0.705 (a ~6.5x performance gain) while remaining completely local, cost-free, and deterministic.

2. Development Tools Used

• VS Code: Primary integrated development environment (IDE) for code implementation, refactoring, and local debugging.

3. APIs Used

• None (Zero External APIs): Although Google's Gemini API was experimentally evaluated for intent-override query rewriting, it was deliberately omitted from the final pipeline due to rate limits and API quota constraints. The production agent runs 100% locally with zero external API calls.

4. Libraries and Frameworks Used

• sqlite3 (FTS5 Extension): Core search backend providing in-memory full-text indexing, BM25 scoring, and customized field weighting.
• re (Python Standard Library): Regular expressions for token extraction (TOKEN_RE), custom stopword filtering, and parsing price constraints.
• json & pathlib (Python Standard Library): Catalog streaming, parsing JSON lines (.jsonl), and cross-platform file path management.

5. Datasets and Assets Used

• Amazon Product Catalog (catalog.jsonl): The provided Amazon dataset containing product metadata, parent ASINs, titles, categories, feature bullet points, details, seller store names, and descriptions. No external third-party datasets or pre-labeled assets were added.

Built With

• bm25
• local
• zero-latency