ClarifyCart

A zero-LLM, fully offline shopping agent that remembers what you say, asks the right follow-up, and finds your exact product in about 3 turns — 92% hit rate on 200 test conversations.

Comment

How our solution addresses the problem statement

The challenge asks for a multi-turn shopping agent that finds a customer's hidden target product as quickly and as highly-ranked as possible, within a hard limit of 10 turns — routing between decisive "buying" and vague "browsing" behavior, handling mid-conversation changes of mind, and asking useful clarifying questions.

Our agent addresses this with four cooperating, fully rule-based components:

  • Intent routing (buying vs. browsing). From the opening message, the agent decides whether the customer arrived with a firm requirement or is still exploring, and adjusts both its catalog search emphasis and how aggressively it asks follow-up questions.
  • Dialogue state machine (accumulate + override). The agent keeps a running "need-notebook," accumulating everything useful the customer says — category, hard constraints, preferences given in answer to a question — into weighted buckets that are reused on every later turn. If the customer changes their mind, the superseded preference is demoted rather than deleted, so a late correction never discards still-useful signal.
  • Proactive clarification. Instead of guessing silently, the agent asks one targeted follow-up per turn through a structured ask_attribute field, leading with a broad probe and then falling back to specific attributes — turning otherwise-wasted turns into new search signal.
  • Rule-based reranking. After retrieving a candidate pool, the agent re-scores products by how fully they satisfy the accumulated notebook — rewarding breadth (satisfying more requirements) and precision (satisfying each one more completely) — blended with the original keyword rank.

Measured with the official evaluator on the 200 public development sessions, this moves the composite technical score from the weak-BM25 starter's 0.10671 to 0.803959.

Overall

Metric Weak BM25 baseline Our submission
Composite 0.10671 0.803959
Hit Rate@10 0.125 0.920
MRR 0.068034 0.615863
MTTC (turns) 9.81 3.04

By scenario (our submission)

Scenario Hit Rate@10 MRR
Buying 0.9125 0.590198
Browsing 0.925 0.583904
Intent Override 0.900 0.738148
Boundary 1.000 0.710000

What we also tried: LLM semantic ranking

The official Pipeline Base for this track describes retrieval followed by LLM semantic ranking. Our submission does not include this step, and we want to be transparent about why: we built it, tested it, and rejected it based on real data rather than skipping it untested.

We implemented an optional, feature-flagged re-ranking layer that takes the top candidates from our existing rule-based reranker and asks a locally-hosted small language model (qwen2.5:1.5b-instruct, served via Ollama, no cloud API, no network access) to reorder them. This kept the experiment consistent with our offline design philosophy even while testing an LLM-based approach. It included a timeout and automatic fallback to the rule-based order on any failure, so the core submission was never put at risk while testing.

Run end to end on all 200 public sessions, the results were unambiguous:

Metric Rule-based (submitted) With local LLM re-ranking
Composite score 0.803959 0.785427
Evaluation wall-clock ~15 seconds ~63 minutes

The LLM layer reduced the score and increased runtime by roughly 250x, with the largest quality drop in the Intent Override scenario. We concluded that for this specific task — reordering a short list of candidates against a set of already-extracted, structured constraints — precise deterministic rules outperform a small local model's semantic judgment, at a fraction of the cost and latency. This experiment is documented in full in PROGRESS.md in the repository, including the code, the fallback design, and the verification steps taken before and after.

Development tools used

  • Python 3.10+ (developed and verified on CPython 3.13) as the sole implementation language.
  • Git and GitHub for version control and submission.
  • The official local evaluator (evaluator/local_evaluator.py), used unmodified, as the deterministic scoring harness for every iteration during development.

APIs used

  • None. The solution makes no external API calls and requires no API keys. It uses no LLM API and no third-party web service, and runs fully offline — which also satisfies the organizer's note that final scoring may run with network access disabled.
  • Internally, the agent implements the required Agent API contract (reset and respond) that the evaluator calls.

Libraries and frameworks used

  • Python standard library only — principally sqlite3 (in-memory FTS5 full-text index) for BM25 retrieval. There are no third-party dependencies and no requirements.txt to install.
  • No large language model and no external vector database is used — a deliberate design choice, explained in the README.

Datasets and assets used

  • The organizer's frozen Clothing_Shoes_and_Jewelry catalog of 50,000 products (from Amazon Reviews 2023), via the official participant-kit release. It is read-only and only parent_asin is scored. The catalog is downloaded at setup time and not committed to the repository.
  • The 200 labeled public development sessions (data/public_set.jsonl) — 80 Buying, 80 Browsing, 30 Intent Override, 10 Boundary — used for local evaluation.
  • No external or self-created catalog entries were used.

Built With

  • bm25
  • conversational-ai
  • fts5
  • no-llm
  • offline-first
  • ollama
  • python
  • recommendation-system
  • rule-based-nlp
  • sqlite
  • state-machine
Share this project:

Updates

Submission history