Shopping Copilot - ShopMind

Inspiration

Track 4's brief asks for something narrower and harder than a chatbot wrapper around a search box: an agent that has to route intent (a shopper stating a hard constraint upfront behaves differently from one still browsing), manage a real dialog state across an abrupt change of mind, and know when it has enough information to commit rather than always asking one more question — all under a hard 10-turn cap where an unnecessary clarification directly costs score. That's a genuine information-seeking problem, not a search-relevance demo, and it's what shaped every architectural decision below.

What it does

Extracts slot updates from the shopper's free-text message (category, material, color, size, style, brand, budget, feature, use case). Routes Buying-vs-Browsing intent and updates a persistent dialog state — accumulating slots normally, but detecting and correctly handling an abrupt intent override (the shopper changing their mind mid-session) rather than just appending contradictory constraints. Retrieves candidates from three fused signals — BM25 keyword search, dense embedding similarity (bge-small-en-v1.5), and metadata filtering — combined with Reciprocal Rank Fusion. Reranks the shortlist with a cross-encoder (ms-marco-MiniLM-L-6-v2). Decides whether to ask a clarifying question or commit to recommendations, based on a calibrated entropy measure over the current score distribution — the over-generality gate that keeps the agent from either dumping an undifferentiated pool of results or asking needless questions on an already-converged pool. No LLM call is required anywhere in this path — the always-scored path runs entirely on local, bundled, open-weight models. An optional Claude Haiku listwise-reranking pass activates only when a local API key is present (never assumed for official grading, since the organizer provides none).

How we built it

Python, sentence-transformers for the two bundled open-weight models (bge-small-en-v1.5 dense retrieval, ms-marco-MiniLM-L-6-v2 cross-encoder reranking), NumPy for vector math, and a hand-rolled, dependency-free BM25 inverted index — kept out of an external BM25 library to minimize the offline/dependency footprint. Everything runs against the organizer's own local evaluator harness and the Amazon Reviews 2023 Clothing_Shoes_and_Jewelry category, using the organizer's frozen 50,000-item catalog kit and 200-session public dev set. An optional anthropic + python-dotenv path (Claude Haiku listwise reranker) activates only when a local API key is present — never a hard dependency, and excluded from the shipped submission's own requirements. Development ran phase-by-phase against the organizer's own local evaluator, with every non-trivial change validated on a held-out split before being kept — see Challenges below for why that discipline mattered more than it might sound.

Challenges we ran into

The Buying scenario lagged Browsing/Boundary in hit rate for most of development — several fixes failed before one actually worked, and the difference was measuring the problem first. Early attempts — widening the reranked candidate pool, reweighting the metadata fusion signal, a query-embedding nudge toward accumulated preference, portfolio-style "slate hedging" for high-uncertainty commits — were all honestly ablated, and all either regressed or failed to replicate on held-out data. One, slate hedging, initially looked like a real win on the training split and shipped — then an adversarial code review caught that its held-out validation result was a genuine wash, and that the reasoning used to justify shipping anyway was exactly the kind of post-hoc rationalization our own pre-registered evaluation protocol exists to rule out. It was reversed. What actually moved the needle was building a diagnostic instead of guessing another fix: a small tool that directly measured whether the hidden target ever reached the retrieval candidate pool at all. It found 37.5% of Buying sessions never did — and proved the pipeline's own "Buying-track hard filter" had zero effect on this, because it only ever restricted on category/brand/budget, never on the disclosed material/color/style constraint that actually defines a buying request. Extending it to those attributes, ablated with the same rigor as every failed attempt before it, raised the overall TechnicalScore by 15.9% — the largest single improvement of the project. A measurement-contamination bug in our own evaluation tooling inflated the headline number itself. The evaluation scripts never explicitly cleared a leftover local .env credential before launching the evaluator subprocess, so every "no API key" (guaranteed-path) measurement was silently exercising the optional LLM-reranking booster anyway. Caught only by noticing the reported score didn't match the expected shape of a no-LLM run. Fixed by explicitly clearing the credential in the evaluation subprocess's environment, and every affected number in this writeup was re-measured and corrected. A reminder that a measurement pipeline is part of the system under test, and needs the same scrutiny as the code it's measuring. A code-review process gap of our own making. A batch of automated code reviews appeared to fail outright and were logged as an environment limitation for several hours. It turned out several had actually completed successfully — the verdict was rendering through a channel plain output redirection didn't reliably capture. Once found, all were recovered from the tool's session transcripts and triaged properly; two were genuine bugs that would very likely have shipped unnoticed (a bundled asset path resolving against the wrong working directory, and a build script that could silently produce an incomplete submission).

Accomplishments that we're proud of

TechnicalScore 0.441 on the full 200-session public dev set (Hit Rate@10 = 0.53, MRR = 0.295, MTTC = 6.63), against the organizer's unmodified baseline of 0.107 — roughly a 4.1x improvement, with zero required LLM calls in the scored path. A named, logged adaptive orchestrator — every retrieval-breadth, rerank-depth, and turn-policy decision is explainable after the fact, not a black box. A real train/validation split discipline held to throughout, including reversing our own already-shipped decision (see Challenges) rather than keeping a change that only won on the split we'd tuned it against

What we learned

Two things, both about process more than any single technique. First, ablation discipline only works if it's applied to itself, not just to the feature being tested — the slate-hedging reversal happened specifically because a validation-split "wash" was almost rationalized away as measurement noise instead of a real answer. Second, and more costly in hindsight: several consecutive fix attempts for the Buying gap failed before we tried actually measuring where the gap lived instead of guessing another plausible mechanism. A 20-minute diagnostic script would have saved several of those earlier detours. The most valuable technical habit this project reinforced isn't any specific retrieval technique — it's writing down and honoring a pre-registered accept/reject rule before looking at a result, and building the measurement before reaching for the next fix.

What's next for Shopping Copilot - ShopMind

Apply the same "measure before you fix" approach to the Intent Override scenario, now the weakest remaining one, instead of guessing at heuristics. The material/color/style hard-filter extension is a single regex-match per attribute; a smarter multi-value or fuzzy match could recover more of the remaining ~24% of Buying sessions where the target reaches the pool but still doesn't make the final top-10 — a ranking problem, not recall. A genuine offline optimization loop over a wider hyperparameter space (only a targeted sweep was run here). The one-step "world-model-lite" lookahead question selector was built and correctly declined (didn't clear a higher bar than the existing entropy heuristic); a two-step extension remains a plausible next experiment if the underlying entropy signal itself improves. Three more mechanisms are built and unit-tested but not yet ablated: seeding the multi-interest state from the session's user_profile field, a Bayesian-shrinkage rating/review-count quality boost, and cross-encoder ensembling. Also pending: RRF per-source weight tuning and offline doc2query-style document expansion.