ShoppingBot

Inspiration

Every "conversational shopping" demo we'd seen reached for the same toolbox — embed the catalog, stand up a vector database, put a large language model in the loop — and then spent its effort papering over the latency, cost, and non-determinism that toolbox drags in. The TechJam Challenge 4 setup let us ask a sharper question: what does this task actually require?

The task is a needle-in-a-haystack conversation. A simulated shopper opens with a vague request; each turn our agent returns ten products and asks one follow-up; the session ends the instant the hidden target appears in the ten. When we looked closely at how the shopper's messages were generated, we found something that reframed the whole project: the shopper's constraints are lifted verbatim from the target product's own catalog text. When the shopper says "waterproof," that exact word is in the target's description. Query and document share vocabulary by construction — so precise word-matching isn't merely adequate, it's optimal. That one measured observation is the seed everything else grew from.

What it does

Shopping Copilot is a conversational product-search agent over a frozen 50,000-item Amazon catalog. Each turn it returns ten recommendations and asks a single follow-up question to narrow things down; the session ends the moment the hidden target lands in the list of ten. It is scored on three axes:

$$\text{TechnicalScore} = 0.50\cdot\text{HitRate@10} + 0.30\cdot\text{MRR} + 0.20\cdot\text{Efficiency}$$

$$\text{Efficiency} = \operatorname{clip}!\left(\frac{11 - \text{MTTC}}{10},\, 0,\, 1\right)$$

where MTTC is the mean turns-to-completion and a miss counts as turn 11.

On the official local evaluator (200 development sessions), the agent scores:

Configuration	Score	Hit@10	MRR	MTTC
Starter baseline	0.1067	—	—	—
Our agent (offline, stdlib)	0.9626	1.000	0.967	2.38

All 200 sessions run in about eight seconds, entirely offline, using only the Python standard library — no network, and no LLM on the scored path.

How we built it

We turned the shared-vocabulary insight into a compact, transparent pipeline, each stage justified by measurement:

Category resolution. The opening message contains the target's coarse category verbatim (e.g. "men's running shoes"). Finding that known category name narrows 50,000 products to a bucket of roughly eight — the single strongest signal, and one that survives any rewording of the surrounding sentence.
Constraint extraction. Each message is mined in two layers: a fast regex layer for the simulator's known phrasings, and a robust salience layer that keeps runs of catalog-grounded, informative, previously-unseen words. Because rewording a sentence doesn't change the requirement's own words, the salience layer survives paraphrase.
Conjunctive ranking. Requirements are an AND, scored as a log-product rather than a sum:

$$\text{score}(d) = \sum_{i} \log p_i(d)$$

so a product satisfying every constraint outranks one that merely nails a single constraint — precisely the tie that otherwise costs precision at the top.

Tie-breaking. When constraints can't separate near-identical products, a Maximal Marginal Relevance diversity re-rank plus a query-independent popularity/quality prior reorders the tied cluster.
A "when to speak" policy. Since a session ends at the first hit, surfacing the target at a poor rank locks that rank in forever. Improving rank is worth roughly $13\times$ a saved turn under the scoring formula, so the agent deliberately holds back weak lists and commits only once the evidence is strong — but never so long that it fails to answer.

The scored path ships with zero third-party runtime dependencies — just json, gzip, re, math, array, collections, dataclasses, pathlib, urllib, hashlib. We also built an optional, off-by-default model-augmentation tier (a local, network-free HyDE rewriter and offline query-expansion) for genuinely free-form language — but it never touches the submitted, scored path.

Challenges we ran into

Our best-sounding ideas measured worse. BM25F rescoring ($-0.04$), vector search (rescued 0 of 200), TF-IDF specificity weighting ($-0.002$), evidence-level alignment (fixed 1, broke 9). They failed for the same structural reason: term-frequency methods assume the query was authored independently of the document — false here — so the "cleverness" was fighting the grain of the problem.
The evaluator's non-obvious rules. On intent_override sessions, hits before the override lands score nothing, so a target sitting at rank 1 for three turns is worth zero. Emitting the moment the override arrives felt right and measured worse, because the turn gained was cheaper than the rank surrendered.
A $\pm0.03$ noise floor at $n=200$. Plenty of changes lived inside the noise, so we couldn't trust a raw delta. We had to build the statistical machinery to tell signal from luck.
Accuracy that cancels itself. A more accurate category key suppressed more of the shopper's own words from extraction, and on free text those words carry coverage — so improving one stage silently degraded another until we decoupled them.
The two-scores trap. The benchmark rewards lexical matching; genuine natural language does not. Keeping those measurements strictly separate — and never letting benchmark tuning masquerade as general capability — was a constant discipline.

Accomplishments that we're proud of

$0.1067 \rightarrow 0.9626$ on the official evaluator, at Hit@10 = 1.000 — retrieval is effectively solved; the residual gap is ranking ties that are information-theoretically unfixable.
A submission that is fully offline and dependency-free on the scored path, so the answer to "does this require the network?" is a clean no — a deliberate compliance choice given that final scoring may run network-disabled.
~8 seconds for 200 sessions, pure standard library.
Evidence over intuition as a habit. Every improvement was adopted only when a paired bootstrap confidence interval confirmed the gain, and every rejected idea was recorded with its measurement rather than quietly dropped.
An independent 26-probe natural-language stress set, authored before any wording and never used to tune the graded score — built to find failures, not flatter the number.

What we learned

Measure the problem before you engineer the solution. The single most valuable artifact we produced was the observation that query and document share vocabulary by construction — it invalidated an entire category of "obvious" upgrades and pointed straight at the cheap, correct method.
Negative results are load-bearing. A justified rejection ("vector search rescued 0/200, here's why") is worth as much as a feature, and it stops you re-litigating the same dead end.
Scoring shapes strategy. Once we internalized rank $\approx 13\times$ a turn, "hold back weak lists" stopped being timidity and became the optimal policy.
Two scores, never conflated. Winning a lexical benchmark and handling real free-form language are different problems; honesty about which one a number reflects is what keeps the write-up trustworthy.

What's next for ShoppingBot

Close the natural-language gap. The scored path is optimized for vocabulary that matches the catalog; the frontier is the shopper who uses their own words. The optional model tier (offline HyDE + query-expansion) is where that work lives, and the goal is to lift it without ever compromising the offline scored path.
Constraint retraction and category revision. Today constraints only accumulate and category is resolved once — so "actually, not leather — canvas" or "never mind, women's instead" isn't handled. Adding retraction and mid-session re-resolution is the next correctness step.
A clean backend seam. A protocol-plus-null-implementation boundary so any future LLM backend is a pure augmentation over a complete, valid, network-free Tier-0 result — never a dependency.
Ship the interactive web UI (tools.webapp) as a proper demo surface, so the conversation is something you can watch, not just score.