Conversational E-Commerce Search

A multi-turn shopping search agent that continuously maintains user intent over a frozen product catalog.

Comment

Overview

We built a multi-turn shopping search agent that continuously maintains user intent over a frozen product catalog. It handles newly added constraints, negations, partial modifications, and “no preference” responses, returning the target product’s Top-10 parent_asin values within at most 10 turns.

During development, we found that the official evaluation protocol could be solved reliably with a fully offline, deterministic architecture—but a high score does not necessarily imply strong natural-language interaction capabilities. We therefore built a separate natural-language evaluator to quantify this gap. Based on the results, we positioned DeepSeek as a controlled “language translation layer” instead of allowing the model to take over product recommendations directly.

The Problem We Solved

Conventional keyword search considers only the current utterance and cannot reliably handle multi-turn shopping conversations. A user might first specify a category, then add a budget, reject a color, change the brand, or say, “You decide the color.” Simply concatenating the text from every turn can cause the system to lose valid constraints, incorrectly remove earlier intent, or eliminate the target product from the candidate pool too early.

Our agent explicitly maintains the current category, hard constraints, soft preferences, negative constraints, previously requested attributes, Boundary state, intent epoch, candidate pool, recommendation history, and evidence provenance. All state changes go through StateReducer, while retrieval, filtering, ranking, clarification, and final response generation remain separate.

Solution

1. Solve the Official Task with a Deterministic System

The official evaluator uses relatively structured inputs, making it well suited to an auditable and reproducible offline pipeline. Starting from a weak SQLite FTS/BM25 search baseline, we progressively added:

  • Dynamic routing between Buying and Browsing;
  • Adaptive category retrieval and a structured candidate pool;
  • Constraint-by-constraint filtering, zero-result fallback, and lexical-tail preservation;
  • Dynamic clarification and recommendation-commitment strategies;
  • Provenance-aware partial Intent Override;
  • Boundary handling for “no preference” responses;
  • Output-schema validation and valid product-ID guards.

On the 200-question public set, the final offline agent achieved:

Metric Weak starter Final offline agent
Hit@10 0.125 1.000
MRR 0.068034 0.805024
MTTC 9.81 3.005
Technical Score 0.106710 0.901407

Hit@10 reached 1.0 across all four official scenario categories: Buying, Browsing, Intent Override, and Boundary. This pipeline consumes zero tokens and requires neither network access nor a model API.

This demonstrates that if the sole objective is to maximize the official evaluation score, structured state management, retrieval, ranking, and commitment strategies are already highly effective; an LLM is unnecessary in the main execution path.

2. Measure the Gap in Real Natural-Language Interaction

The official protocol does not adequately cover real-world interactions involving synonyms, negations, multiple constraints, references, or hidden profiles. We therefore created an independent intelligent natural-language evaluator v2 with:

  • 100 frozen test cases;
  • A catalog of 50,000 products;
  • Up to 10 turns per conversation;
  • Exact Top-10 parent_asin scoring;
  • Seven scenario categories: Budget Rating, Clarification, Direct Search, Intent Override, Multi Constraint, Negative Constraint, and Profile Hidden.

How the Evaluator Works

Rather than asking a large model to freely role-play the user, the evaluator is a freezable, verifiable, multi-turn test harness that does not leak the target:

Frozen catalog
   ↓
A deterministic generator selects a target product
and creates a unique fact signature
   ↓
Facts are divided into an initial query,
an anonymous profile, and hidden clarification slots
   ↓
Call the agent each turn → interpret ask_attribute
and natural-language questions
   ↓
Reveal only the next fact supported by the target product
and update candidate state
   ↓
Record a turn-by-turn trace and perform exact
parent_asin scoring at the end

The evaluator follows four key principles:

  1. Select the target before generating the conversation. The generator chooses a target from the local catalog and requires the complete conjunction of facts to uniquely identify it. The initial information intentionally leaves multiple candidates so that the answer is not revealed on the first turn.
  2. Divide facts into known and hidden information. Some facts are included in the initial query or anonymous profile, while the rest are stored in clarification slots. The agent receives the next hidden fact only by asking an appropriate follow-up question or when the user naturally expresses that constraint.
  3. Understand both structured and natural-language questions. The evaluator reads the agent’s ask_attribute field and parses natural-language questions in its message. If the two conflict, the question is marked ambiguous rather than guessed. It also recognizes synonyms, repeated questions, overly broad questions, no-preference responses, irrelevant questions, and exhausted information.
  4. Keep the evaluator isolated from the agent. target_parent_asin, fact signatures, and candidate counts exist only in the parent process and scoring system; they are never passed to the agent. Each turn records the intent path, state, retrieval and ranking stages, and question interpretation through a side channel. This makes it possible to distinguish failures caused by misunderstanding, safety-validation rejection, retrieval omissions, or commitment-strategy errors.

We generated and froze 100 test cases using a fixed seed. A validator then rechecks that every query, profile, override, and hidden response genuinely contains its corresponding facts. As a result, every comparison uses identical inputs, and changes in DeepSeek’s performance can be attributed to the language-understanding layer rather than random variation in the questions.

In the implementation, generator.py generates targets and fact signatures from the catalog; simulator.py advances multi-turn user state; question_interpreter.py interprets the agent’s questions; validator.py prevents target leakage and fact inconsistencies; and evaluator.py and metrics.py run the agent in isolation and compute the metrics. Evaluation uses a disposable worker subprocess, ensuring that the parent process never imports target fields into the agent’s call stack.

The frozen set can be validated and evaluated with the following commands, run from the separate benchmark repository:

cd /path/to/techjam-natural-language-benchmark
python3 -m nl_benchmark validate \
  --catalog /path/to/techjam-conversational-search-main/data/catalog.jsonl \
  --dataset outputs/frozen-100-seed-20260830.jsonl

python3 -m nl_benchmark evaluate \
  --protocol-profile natural_language \
  --agent-repo /path/to/techjam-conversational-search-main \
  --catalog /path/to/techjam-conversational-search-main/data/catalog.jsonl \
  --dataset outputs/frozen-100-seed-20260830.jsonl \
  --output /tmp/techjam-natural-language-results.json

Scoring still focuses exclusively on the final task: whether the target product appears in the Top 10, its rank on the first successful hit, and the turn on which it first appears. No credit is awarded based on model self-evaluation or textual similarity. The intent_override scenario also explicitly constructs a transition from an old, incorrect preference to a new target fact. Recommendations made before the override do not count toward the exact metrics for the new intent.

When we placed the first version of the deterministic agent directly into this evaluator, it achieved a Hit@10 of only 0.580 and an average MTTC of 7.08. This does not mean the official agent had “failed”; rather, it shows that a template-driven, high-scoring pipeline has limited coverage of natural-language expression.

3. Use DeepSeek as a Controlled Language Translation Layer

We added a DeepSeek canonicalizer that rewrites colloquial requests into “shopping facts” the deterministic system can reliably process. The model does not return product recommendations. Instead, it returns JSON containing only canonical_text, with each line expressing exactly one constraint.

For example, the user might say:

I'm after something comfy for rainy commutes. Keep it under $80, and please don't show me black. Actually, make it Skechers instead of the brand I mentioned earlier.

DeepSeek converts this into:

{
  "canonical_text": "A key requirement is: use_case: rainy commutes.\nA key requirement is: feature: comfortable.\nA key requirement is: budget: under $80.\nI do not want color: black.\nActually, change the brand to Skechers."
}

The existing deterministic parser then converts these lines into constraints that the agent can execute:

  • Use case: rainy commutes;
  • Desired feature: comfortable;
  • Budget: no more than $80;
  • Excluded color: black;
  • Brand: replace the previous brand with Skechers while preserving all other constraints.

The translation layer may describe only fields that are genuinely useful for shopping, including category, brand, budget, minimum rating or review count, color, material, size, style, features, and use case. It also supports negations, removals, and partial modifications such as “no black,” “budget no longer matters,” and “change the brand to Skechers.”

Natural-language message
  → DeepSeek converts it into one shopping fact per line
  → Validate that the facts preserve the user’s intent
    and exist in the product catalog
  → Update only the preferences that the user explicitly
    added or modified
  → Pass the result to the existing retrieval, ranking,
    clarification, and recommendation pipeline

To prevent situations where the model produces fluent but incorrect interpretations, the system performs several validations on every turn. Values such as brands, colors, and materials must exist in the local product catalog. The translation may not contain product IDs or fields unknown to the system. If the user only asks to change the brand, the model may not also clear the budget or category. Preferences that the user has not explicitly withdrawn must remain in effect.

If any validation fails, the entire model translation for that turn is discarded and the system falls back directly to the original offline rule-based result. This prevents partially correct and partially incorrect information from contaminating subsequent turns.

Results

DeepSeek produced substantial improvements on the independent natural-language evaluator v2:

Metric Deterministic agent + DeepSeek translation layer Change
Hit@10 0.580 0.770 +19 percentage points
Exact Top-1 0.270 0.510 +24 percentage points
MRR 0.365913 0.605179 +0.239266
MTTC 7.08 3.72 −3.36 turns

The largest gains appeared in Direct Search (0.800 → 1.000), Intent Override (0.429 → 0.857), Profile Hidden (0.143 → 0.500), and Negative Constraint (0.643 → 0.786). Multi Constraint declined from 0.867 to 0.800, indicating that the model layer still requires stricter validation of multiple simultaneous constraints.

The two evaluators use different samples and protocols, so their results should not be combined into a single leaderboard. They answer two different questions: the official evaluator measures reliable task completion within the protocol, while the natural-language evaluator measures robustness to linguistic variation.

Technology Stack, Tools, and APIs

  • Development tools: VS Code, the Python 3.10+ command line, and unittest. The repository retains a Jupyter/Colab-style notebook for Qwen reranker experiments.
  • Core libraries: Python standard-library modules including json, re, sqlite3, and pathlib. SQLite FTS5/BM25 is used for offline retrieval.
  • Optional experimental dependencies: The Qwen cross-encoder path may use sentence-transformers and PyTorch. It is not part of the official offline execution path.
  • External API: DeepSeek V4 Flash through an OpenAI-compatible API, used only for translating natural-language constraints. The API key is injected through an environment variable and is never stored in the repository.
  • Runtime boundaries: The official submission can use its deterministic fallback with network access disabled. The DeepSeek version is an optional enhancement path.

Reproduction

Offline Version: Main Official Evaluation Path

No model credentials are required. The following command reproduces the deterministic agent on the official evaluator:

python3 -m evaluator.local_evaluator \
  --protocol-profile official \
  --catalog data/catalog.jsonl \
  --dataset data/public_set.jsonl \
  --output results/techjam-official-offline.json

Online Version: DeepSeek Natural-Language Enhancement

First add SHOPPING_AGENT_DEEPSEEK_API_KEY to a local .env file, then explicitly enable the natural-language model layer. Do not place the key in commands, source code, or the submission package:

set -a; source .env; set +a
export SHOPPING_AGENT_INTENT_MODEL_ENABLED=true
export SHOPPING_AGENT_INTENT_MODEL_MODE=model_first
export SHOPPING_AGENT_MODEL_TIMEOUT_SECONDS=30

python3 -m evaluator.local_evaluator \
  --protocol-profile natural_language \
  --catalog data/catalog.jsonl \
  --dataset data/public_set.jsonl \
  --output results/techjam-official-offline.json

Both commands use the same official public dataset. The 100-question frozen set for natural-language evaluator v2 is run from a separate benchmark repository.

Natural-Language Evaluator v2

The v2 evaluator is located in a separate repository:

https://github.com/deequoique/techjam-natural-language-benchmark

Requirements

  • Python 3.10+
  • The complete data/catalog.jsonl
  • The frozen test set at outputs/frozen-100-seed-20260830.jsonl
  • No third-party Python dependencies
git clone https://github.com/deequoique/techjam-natural-language-benchmark.git
cd techjam-natural-language-benchmark

python3 -m unittest discover -s tests -v

First, validate the frozen 100-question dataset:

python3 -m nl_benchmark validate \
  --catalog /path/to/techjam-conversational-search/data/catalog.jsonl \
  --dataset outputs/frozen-100-seed-20260830.jsonl

Test the Deterministic Agent

python3 -m nl_benchmark evaluate \
  --protocol-profile official \
  --agent-repo /path/to/techjam-conversational-search \
  --catalog /path/to/techjam-conversational-search/data/catalog.jsonl \
  --dataset outputs/frozen-100-seed-20260830.jsonl \
  --output /tmp/nl-v2-offline.json

Test the DeepSeek-Enhanced Natural-Language Agent

set -a
source /path/to/techjam-conversational-search/.env
set +a

export SHOPPING_AGENT_INTENT_MODEL_ENABLED=true
export SHOPPING_AGENT_INTENT_MODEL_MODE=model_first
export SHOPPING_AGENT_MODEL_TIMEOUT_SECONDS=30

python3 -m nl_benchmark evaluate \
  --protocol-profile natural_language \
  --agent-repo /path/to/techjam-conversational-search \
  --catalog /path/to/techjam-conversational-search/data/catalog.jsonl \
  --dataset outputs/frozen-100-seed-20260830.jsonl \
  --output /tmp/nl-v2-deepseek.json

After evaluation, the summary metrics are printed directly in the terminal. Complete results and turn-by-turn traces are saved respectively to:

  • /tmp/nl-v2-offline.json
  • /tmp/nl-v2-deepseek.json

Interactive CLI

# Offline version
python3 -m starter.cli --protocol-profile official

# Online DeepSeek version
set -a; source .env; set +a
export SHOPPING_AGENT_INTENT_MODEL_ENABLED=true
export SHOPPING_AGENT_INTENT_MODEL_MODE=model_first
python3 -m starter.cli \
  --protocol-profile natural_language \
  --show-diagnostics

Limitations and Next Steps

  1. The natural-language evaluator remains a frozen benchmark and does not represent the full distribution of real online users. The next step is to expand coverage of linguistic variation and conduct cross-domain testing.
  2. DeepSeek adds latency and token costs. Running the 100-question set took 677.93 seconds and reported 940,437 tokens. The cost is an estimated range based on caching assumptions rather than an invoiced amount.
  3. The current model layer focuses primarily on intent translation. It does not yet cover personalized explanations, long-term preference learning, or multi-product comparisons.
  4. Our next priority is a rules-first approach with low-confidence model triggering, together with complete logging of usage and cache-hit information for failed requests.

Submission Links

Team Contributions

  • Tonglu Wang: Led the overall architecture and implementation of the project; proposed and implemented the LLM-assisted natural-language interaction approach; led the design and development of the flexible evaluator; and completed most of the coding, project coordination, testing, and documentation.
  • Yujia Tian: Optimized the offline agent, increasing the Technical Score from approximately 0.80 to 0.90.
  • Siyu Ren: Conducted agent-loop architecture experiments, testing, and solution research.
  • Wenjie Luo: Built the benchmark dataset and optimized evaluator testing.
  • Zedong Jiang: Organized project concepts and contributed to solution discussions and collaborative testing.

Built With

Share this project:

Updates

Submission history