Description
The challenge asks for a multi-turn shopping agent that identifies a customer's intended product from a 50,000-item Amazon catalog in as few turns as possible. Our core finding was that retrieving on the opening message alone surfaces just 1.7% of targets in the top 10, while retrieving on everything a shopper discloses across the conversation surfaces 85%. The bottleneck was extraction and retention of what the customer says — not the ranking algorithm — and the whole architecture follows from that.
Why this matters beyond the benchmark. The provided baseline takes 9.81 turns to converge; we take 1.875. That is roughly eight fewer questions a shopper answers before seeing the right product. In a real storefront that difference is the gap between a search that feels like a conversation and one people abandon halfway through. The same mechanism — ask about what's actually missing, never forget an answer, react when someone changes their mind — is exactly what conversational commerce needs to work at scale, and none of it requires a model call per turn.
Every turn recommends and asks at the same time. We return recommendations and ask about the single highest-yield unanswered attribute, ordered by measured yield: feature 96%, material 76.5%, colour 25.5%, style 9%, size 4.5%, use_case 2%, with budget/brand/category at 0% and excluded from questioning entirely. This is right on two counts. The evaluator scores recommendations before it reads the question, so asking is free — and in a real store, showing results while you ask beats a blocking question that stalls the shopper. That reframed the design problem: not when to ask, but when to stop, which we do once the leading candidate is confidently ahead of the field.
State accumulates across turns, with one exception. An Intent Override — the shopper contradicting an earlier preference — retracts only that specific constraint, never the whole session. An earlier version that wiped everything on any override cost us both hit rate and turns. A Boundary reply ("no preference for colour") permanently settles that attribute so it is never asked again; re-asking burns a turn and can never yield anything.
Ranking rewards constraint coverage — how many stated requirements a candidate verifiably satisfies — because 97.1% of disclosed constraints appear verbatim in the target's own listing. Popularity (review volume as a purchase-likelihood proxy) and category-path match act as secondary signals, and the long-term user profile is consulted only to break exact ties, because measured as an additive term it made ranking worse.
Results. TechnicalScore 0.107 → 0.876, an 8.2× improvement, with 197 of 200 public sessions finding the target. Reported per scenario, as the evaluator requires: Buying reaches 0.988 Hit Rate@10 with 0.691 MRR in 1.36 turns; Browsing 1.000 with 0.591 MRR in 1.46 turns; Intent Override 0.967 with 0.815 MRR in 3.97 turns — high by construction, since the evaluator cannot record a hit before the changed intent is revealed; and Boundary 0.900 with 0.692 MRR in 3.00 turns.
Feasibility. All 200 sessions run in 18.0s — 59ms per session, 32ms per turn — with a one-off 6.2s catalog index build and 638 MiB peak memory, on Python 3.10.12. Token usage is 0 and model cost is $0.00, because no model is called. There are no credentials to configure, no rate limits, and no service that can be down when someone re-runs it.
What we measured and rejected. An LLM reranking stage via the Groq API improved ambiguous cases (+0.041 MRR) but demoted already-correct ones (−0.107 MRR) across 40 real sessions, projecting to a net −0.011 TechnicalScore even under a best-case invocation gate. We kept the experiment in the repo as evidence rather than deleting it. Title-weighted matching also lost (0.876 → 0.854), and intent-conditioned retrieval weighting turned out to be a rigorous null — both Buying and Browsing were already at their optimum on the shared weights — so we report it as neutral rather than dressing it up as a gain. Note that the rules permit network access and do not require an offline fallback: running offline is a deliberate engineering choice we measured our way into, not a constraint we were handed.
Limitations. Constraint parsing is marker-based, so a paraphrase outside the observed disclosure phrasings yields nothing; a small extraction model is the obvious upgrade. Attribute matching is bound to the evaluator's fixed colour/material vocabulary. Boundary is our weakest scenario at 0.900, though with only 10 such sessions the estimate is noisy. A learned ranker is the right next experiment for the remaining MRR headroom, but with only 200 public sessions we judged the overfitting risk to outweigh the upside without proper cross-validation.
Reproducibility. python3 -m evaluator.local_evaluator prints recommended_technical_score: 0.875866 against the unmodified official evaluator. All reported figures come from commit 6f5e0f8; the README documents the exact hardware, Python version, measurement method for every number, and both catalog checksums.
Development tools: Visual Studio Code, Git/GitHub, pytest, the official TechJam evaluator harness, and AI coding assistance (Claude) for implementation and review.
APIs used: None in the final pipeline — fully offline and deterministic, with no network calls at inference time. We evaluated the Groq API for LLM reranking (scripts/validate_llm_rerank.py) and rejected it on measurement, as above.
Libraries and frameworks: Python standard library only on the scored path — retrieval runs on sqlite3's built-in FTS5 full-text index, entirely in memory, with zero third-party runtime dependencies (verified by walking the import graph of agent.py, config.py and src/). groq and python-dotenv are isolated in requirements-dev.txt and needed only to re-run the rejected experiment; nothing in the scored pipeline imports them.
Datasets and assets used: Amazon Reviews 2023 (McAuley Lab, UCSD), source of both the 50,000-item catalog and the simulated shopper sessions (see DATA_ATTRIBUTION.md). Used read-only and unmodified.
Team: Yong Chuan Onn, Chew Qiao Enn, Balon Alexandre Stephane Daniel, Stella Teo Boon Yim.
Log in or sign up for Devpost to join the conversation.