Conversational Shopping Copilot

An adaptive, CPU-local shopping agent that routes browsing and buying differently, tracks evolving context, and finds the target in 199/200 public sessions—with zero APIs, tokens, or GPUs.

Comment

Inspiration

Most shopping searches do not begin with a complete specification. A customer may start by browsing, gradually add constraints, reject an attribute, ask for alternatives, or change direction entirely.

Traditional keyword search treats each message as an isolated query. We wanted to build something different: a shopping agent that understands how intent evolves, preserves useful evidence across turns, asks a question only when it can meaningfully reduce uncertainty, and changes its retrieval strategy at runtime.

Our guiding principle was simple:

A shopping assistant that asks the right question is more useful than one that returns a longer list.

What it does

Conversational Shopping Copilot is a local, multi-turn shopping agent designed to place a simulated shopper's hidden purchased product in the Top 10 within ten turns.

On the official public evaluator, the submitted score_default configuration achieved:

Metric Result
TechnicalScore 0.932067
Hit Rate@10 0.995 — 199 of 200 sessions
MRR 0.852556
Mean Turns to Conversion 2.06 turns
Warm-turn latency 25.571 ms p50 / 30.445 ms p95
Runtime API calls / tokens / cost 0 / 0 / $0

This is an 8.7× improvement over the provided weak baseline TechnicalScore of 0.10671.

The scored configuration runs entirely on a local CPU using the Python standard library. It requires no GPU, external API, model server, vector database, credentials, or network access.

How we built it

1. Intent routing and a multi-route retrieval pipeline

Every turn is classified into one of four states:

  • Browsing — preserve diversity when intent is still open-ended
  • Buying — prioritize precision when usable constraints are available
  • Mixed — combine exploratory and targeted signals
  • Override — recognize when the customer changes direction

The route can evolve during the same conversation. A session may begin in Browsing and automatically retarget toward Buying as actionable evidence accumulates.

The scored retrieval pipeline combines:

  • SQLite FTS5 keyword retrieval
  • category and facet signals
  • typed slot matching
  • popularity and phrase evidence
  • deterministic candidate funnels
  • bounded rescue candidates
  • feature-based reranking

The repository also includes an in-memory dense candidate source with reciprocal-rank fusion and a quantized TinyBERT semantic Top-10 reranker. Additional dense, semantic, profile, and override-policy capabilities are documented in the feature-off section below.

2. Multi-turn dialogue evolution

The agent does not store conversation history as a flat bag of words. Each extracted constraint becomes a typed record containing:

  • attribute and value
  • positive or negative polarity
  • hard or soft preference
  • confidence
  • source turn
  • provenance
  • active and credibility state

This supports incremental information accumulation, negation, contradiction handling, abandoned evidence, and abrupt intent changes.

When the live candidate pool remains too broad, the agent stops asking a generic question. It measures the current pool and selects the attribute that best separates the remaining products, then presents structured options to help the customer converge.

When a customer provides little new evidence or asks to see alternatives, the retrieval controller can widen candidate depth from 100 to 1,000 while preserving confident recommendations and refreshing the tail.

3. Runtime context programming

Retrieval and dialogue behavior are controlled by bounded, immutable context snapshots derived from the current session.

A pre-retrieval controller decides whether the turn should use standard or widened retrieval. A question controller decides whether clarification is appropriate, which attribute should be asked, and whether the prompt should be open or structured.

These controllers are part of the shipped default. They were first evaluated in shadow mode and adopted only after matching the previous behavior on every compared turn.

The project also implements bounded profile normalization and profile-credibility analysis. Profile signals can be classified as unsupported, generic, or informative relative to the live candidate pool. Profile ranking remains off in the scored configuration because the official profiles did not demonstrate reliable target-discriminating information.

Optional capabilities and why they remain feature-off

The architecture includes several independently selectable capabilities beyond the submitted score_default. Each one is implemented, tested, and demonstrable, but remains off unless its measured benefit justifies its additional cost or risk.

Optional capability What it demonstrates Why it is off in score_default
Dense retrieval + reciprocal-rank fusion A deterministic in-memory dense source retrieves cross-category candidates that BM25 cannot reach, then fuses lexical and dense ranks through RRF. It added genuine candidate diversity and improved some Browsing results, but the measured trade-off did not pass the global quality and robustness gates.
TinyBERT A2-10 semantic reranker A 4.5 MB quantized cross-encoder semantically reorders a copy of the visible Top 10 on eligible early Browsing turns. It preserves the recommendation set, so HR@10 and MTTC remain invariant. It passed the frozen CPU feasibility gates and improved supplementary-validation MRR by approximately 0.008, but it has no official public result under our one-finalist confirmation protocol. It also introduces ONNX dependencies, latency, and memory cost.
Profile credibility and ranking External preference tags are normalized and classified relative to the live candidate pool as unsupported, generic, or potentially informative. The official profiles did not demonstrate reliable target-discriminating information. Credibility telemetry is available, but profile ranking weight remains 0.0.
Targeted override erasure An alternative dialogue policy can erase only evidence superseded by an intent pivot instead of retaining or clearing the full history. It is implemented as a configuration option, but the measured default retention policy produced the stronger overall result.

This feature-off discipline is intentional. Optional components remain available for architecture demonstrations and future datasets, while the scored configuration stays fast, dependency-free, and supported by the strongest reproducible evidence.

None of these optional capabilities contributes to the reported TechnicalScore of 0.932067.

4. Evaluation as an engineering component

Evaluation integrity is part of the architecture, not an afterthought.

Every experiment records its:

  • exact Git commit
  • complete configuration
  • seed list and per-seed metrics
  • clean or dirty repository state
  • dataset, catalog, scenario, and agent hashes
  • completion status and provenance

Experiments run under exclusive leases in detached Git worktrees. Results are written to append-only ledgers and are never silently overwritten or deleted. Invalid results remain visible with explicit invalidation reasons, and one citability predicate determines which rows may be quoted.

The final release contains 828 tests, including exact score locks, configuration locks, controller equivalence grids, negative controls, input-bound checks, provenance validation, and clean-checkout reproduction.

The same TechnicalScore of 0.932067 was independently reproduced on Ubuntu with both Python 3.10 and Python 3.11 through GitHub Actions.

Challenges we ran into

The hardest engineering problem was not retrieving more products. It was deciding which evidence remained trustworthy as the conversation evolved.

A stated constraint may later be contradicted. A vague request may not contain enough evidence for a narrow search. A customer asking for more options needs broader retrieval, but broad retrieval on a strong query can introduce popular near-misses.

We addressed this by making confidence, polarity, hardness, abandonment, candidate depth, and clarification strategy explicit parts of the runtime state rather than hidden heuristics.

We also found that measurements could become unreliable if an experiment read a repository while another process was changing it. That led us to build exclusive experiment leases, immutable worktrees, file fingerprints, append-only result ledgers, and fail-closed reporting.

Accomplishments that we are proud of

  • Reaching the exact target in the Top 10 for 199 of 200 official public sessions
  • Improving the weak baseline TechnicalScore by 8.7×
  • Guiding successful sessions to the target in 2.06 turns on average
  • Running the scored path with zero external dependencies and zero runtime API cost
  • Building explicit, inspectable controllers for both retrieval depth and clarification
  • Implementing all four competition pillars in a modular architecture
  • Preserving experimental failures and invalid measurements instead of removing them
  • Reproducing the score from a clean checkout and on two Linux Python versions

What we learned

Better conversational search is not simply deeper retrieval.

The strongest improvements came from retaining trustworthy evidence, asking candidate-aware questions, and adapting retrieval only when the session needed it. A feature that sounds sophisticated should not enter the scored path merely because it exists; it should earn that position through reproducible evidence.

We also learned that evaluation infrastructure can be as important as model architecture. A score is much more valuable when every step used to produce it can be independently inspected and reproduced.

Practicality and impact

A zero-cost, CPU-local shopping agent can be deployed per session without reserving a model-serving budget for every visitor. It can operate in privacy-sensitive or offline environments, and its decisions can be inspected through explicit route, retrieval, evidence, and question reason codes.

The current results are limited to the official English-language sessions and a 50,000-product Clothing, Shoes and Jewelry catalog. We make no claim about the private evaluation set, other languages, or other retail verticals.

What's next

Future work would include:

  • evaluation on additional retail categories and languages
  • stronger enforcement guarantees for explicit hard constraints
  • profile testing with stable user identity and genuinely discriminative history
  • architecture-specific semantic artifacts for platforms beyond Apple silicon
  • validation on a public-like corpus without the official dataset's popularity bias

Technology and resources

The scored path uses Python and SQLite FTS5 from the standard library.

Optional showcase components use ONNX Runtime, NumPy, Tokenizers, and the Apache-2.0 cross-encoder/ms-marco-TinyBERT-L2-v2 model. The project code and documentation are MIT licensed.

The official catalog contains 50,000 products derived from Amazon Reviews 2023 and distributed by the organizer. The official public development set contains 200 sessions. We also generated clearly labelled synthetic supplementary sessions for robustness research; they are never presented as official or real-user results.

Claude Code and OpenAI Codex were used as engineering and independent-review assistants under the participant's direction. The scored agent itself uses neither service and makes no external runtime calls.

Links

Built With

Share this project:

Updates

Submission history