TokTik's Shopping Copilot

Shopping Copilot: AI Conversational Search and Recommendations

TokTik's Shopping Copilot is a deterministic, offline conversational shopping agent that helps customers search a 50,000-product fashion catalogue. The challenge is to identify a hidden target product within ten conversation turns, even when the customer starts with a vague request, changes their mind, or declines to provide a preference.

On every turn, our agent converts the conversation into structured preferences across nine attributes: category, material, colour, size, style, brand, budget, feature, and use case. It then dynamically classifies the current request as either Buying or Browsing and selects an appropriate retrieval strategy.

The retrieval pipeline combines exact and Porter-stemmed SQLite FTS5 BM25 search, semantic retrieval using a locally executed MiniLM model, and exact-phrase matching. Reciprocal Rank Fusion combines these independent rankings without mixing incompatible raw scores. Results are then reranked using customer constraints, catalogue-source confidence, budget, popularity, and safe aggregate profile tags.

The agent also maintains conversation state. It understands negation and “no preference” responses, preserves constraints across turns, and retracts obsolete preferences when the customer changes their intent. Products already shown under the current intent are treated as implicit negative feedback and are not recommended again. When an intent override occurs, this exclusion history is reset because earlier recommendations addressed a different need.

Clarification questions are selected using expected information gain over the remaining candidates. An unproductive answer encourages the agent to ask a specific follow-up question, while repeated silence or refusals retire the open-ended action for the current intent. The policy also limits consecutive open-ended questions and stops asking once all useful topics are exhausted. This prevents repetitive dialogue while retaining the ability to gather preferences that do not fit a predetermined attribute.

What was provided by the organisers

The initial repository contained:

• A frozen catalogue of 50,000 products derived from the Clothing_Shoes_and_Jewelry category of Amazon Reviews 2023, distributed through the repository’s GitHub Release.
• 200 labelled public development sessions: 80 Buying, 80 Browsing, 30 Intent Override, and 10 Boundary sessions.
• A weak, stateless SQLite FTS5 BM25 starter agent.
• The deterministic local evaluator, Agent API contract, evaluation configuration, scoring rules, data attribution, and submission requirements.
• Baseline results of 0.125 Hit Rate@10, 0.068034 MRR, 9.81 MTTC, and a 0.10671 TechnicalScore.

We retained the organiser’s evaluator and public labels unchanged. Our team replaced the editable starter agent and created the understanding, retrieval, dialogue-policy, orchestration, artifact-building, benchmarking, interactive testing, and robustness components.

Development tools

• Visual Studio Code
• Python 3.10+
• Git and GitHub, using feature branches and pull requests
• Terminal-based development and benchmarking
• The organiser’s deterministic local evaluator
• Python unittest
• Custom command-line tools for index building, metric sweeps, failure analysis, robustness testing, and interactive conversations

APIs and external services

The scoring path does not call OpenAI, GPT-4o, or any other cloud inference API. It implements the organiser’s local reset/respond Agent API and runs entirely offline after setup.

Hugging Face is used only during the initial artifact build to download a pinned ONNX export and vocabulary for sentence-transformers/all-MiniLM-L6-v2. Runtime evaluation reports zero prompt and completion tokens and has an estimated API cost of $0.

Libraries, frameworks, and models

• SQLite FTS5 for exact and Porter-stemmed BM25 retrieval
• NumPy for dense-vector storage and cosine similarity
• ONNX Runtime 1.22.1 for local MiniLM inference
• sentence-transformers/all-MiniLM-L6-v2, using its 384-dimensional ONNX export
• Python standard-library modules including re, dataclasses, concurrent.futures, sqlite3, and unittest
• A custom WordPiece tokenizer and Reciprocal Rank Fusion implementation
• We deliberately avoided PyTorch, Transformers, and Sentence Transformers runtime libraries, keeping inference comparatively lightweight and CPU-friendly.

Datasets and assets

• Amazon Reviews 2023 by McAuley Lab, UCSD
• The organiser’s 50,000-product frozen clothing, shoes, and jewelry catalogue
• The 200 labelled public conversational development sessions
• Aggregate, anonymised profile tags supplied with each session
• The pinned MiniLM ONNX model and vocabulary from Hugging Face
• Team-created synonym lexicons and catalogue attribute mappings
• A generated provenance-aware attribute index
• A manually authored adversarial test corpus containing 332 single-message cases and 31 multi-turn dialogues covering negation, overrides, prices, sizes, idioms, vague requests, and boundary behaviour

Results

Using the untouched local evaluator on all 200 public sessions, our final system achieved:

The agent found the hidden target in 198 of the 200 public sessions. Compared with the organiser-provided starter agent, it improved Hit Rate@10 from 12.5% to 99.0%, increased MRR from 0.0680 to 0.6530, and reduced mean time to conversion from 9.81 turns to 2.655 turns. TechnicalScore increased from 0.1067 to 0.8578.

The public simulator sometimes returns structured catalogue-derived language, so we separately tested natural phrasing and adversarial inputs. The current implementation passes 100% of the hard robustness cases. Its primary limitation is that preference extraction remains deterministic, meaning uncommon paraphrases, brands, fashion subcultures, and non-US sizing conventions may still be missed.

Built With

• numpy
• onnx
• python
• sqlite