ShopSage

Inspiration

Every shopping search box assumes you already know what you want. Real shoppers don't — they say "something comfortable for work" and then discover, four questions later, that what they actually meant was a wide-fit leather loafer under $80. The TechJam challenge captures that exactly: a hidden target product, a vague opening message, and at most 10 turns to get it into the top 10.

What pulled us in was a detail in the scoring rules. Reciprocal rank freezes the moment the target first appears in a returned list, so an agent that dumps ten guesses on turn 1 permanently banks a bad rank. That turned the obvious strategy on its head. Knowing when to stay quiet and ask instead becomes a ranking decision — and that idea ended up shaping the whole agent.

What it does

ShopSage is a multi-turn conversational shopping agent over a frozen 50,000-product Amazon Clothing_Shoes_and_Jewelry catalogue. Each turn it takes the shopper's message plus an anonymized preference profile and returns three things: a natural clarification question, one structured ask_attribute the simulator can answer, and a ranked list of parent_asin values.

The interesting behaviour is that the list length is not fixed. Every turn the agent measures how far its leading candidate has separated from the runner-up, as a fraction of the top score, and sizes the response to that confidence:

Separation	Leader really is the target	Results returned
≥ 0.30	97.9%	10
0.20 – 0.30	92.3%	10
0.10 – 0.20	66.7%	2
< 0.10	32.0%	1

Below the threshold it asks a question instead of guessing. From turn 5 the list always widens to 10, because never answering costs the full 11-turn miss penalty.

Measured on the official evaluator:

	Technical score	Hit@10	MRR	MTTC
Public set (200 sessions)	0.9143	0.990	0.9006	3.55
Held-out set (200 unseen targets)	0.8766	0.955	0.8525	3.83

The provided BM25 starter scores 0.1067.

How we built it

Retrieval. Eight lexical routes over SQLite FTS5 (BM25) — latest message, attribute, exact phrase, identifier, feature/use-case, relaxed, category, synonym — each weighted toward different catalogue fields and fused by weighted reciprocal rank, plus two vector routes (BAAI/bge-small-en-v1.5, prebuilt index) that carry recall when lexical evidence is thin. Fused ranks are then combined with additive evidence: confirmed preferences, exact phrase and identifier matches, category agreement, a penalty for matching a rejected value, and a multi-route agreement bonus.

Every constraint value is IDF-damped before it scores, so catalogue boilerplate can't outweigh distinctive evidence — Imported counts 0.54, 100% Cotton 0.71, a model number 1.00.

Buying and browsing take different paths. Intent is classified per turn. On the buying path, constraints are typed by how the evidence arrived: volunteered or corrected is hard, an answer to a question we asked is soft. Hard constraints filter the pool rather than nudging it, with two safeguards — a product is dropped only when it can be shown to violate, so missing metadata never excludes anything, and the filter surrenders its least reliable constraint when survivors fall below a floor rather than starving. On the browsing path nothing is filtered; vector recall does more work and the agent converges by asking.

Dialogue state. Slots accumulate across turns, negations are tracked separately from preferences, and an intent override erases and rewrites rather than merging. The open-ended question is promoted the moment structured facets stop working — right after an override, or once the shopper has declined two attributes.

The LLM is gated, not central. A deterministic parser handles state interpretation; gpt-5.6-luna (via LangGraph with a forced structured tool call) is routed only the turns the parser can't read confidently — about 32 of 800 turns. That's ~51,000 tokens per 200-session run versus 1,027,203 ungated, a 95% reduction for a score that doesn't move. The model never generates product IDs, and every LLM path falls back to the deterministic parser on failure. With OPENAI_ENABLED=false the agent runs fully offline with no network calls at all.

Guardrails. An audience guardrail penalises department mismatch — the catalogue skews 2.4:1 toward women's, so men's and boys' queries were being outranked by products matching everything except the shopper. Wrong-audience results in the top ten fell from 16.6% to 7.9%. Recommendation explanations are generated deterministically from recorded match evidence, never from a model, so the agent can't claim a preference the product doesn't have.

Per turn: p50 168 ms, p95 277 ms, p99 455 ms (no API call). 338 tests.

Challenges we ran into

Most of our good ideas were wrong, and we could only tell by measuring. We built and rejected nine features, each measured on both the public and held-out sets:

Rejected	Public	Held out	Why
Dual-track weight multipliers	−0.0017	−0.0066	bought MRR by shedding Hit@10
Result diversity	−0.0134	—	spreading pushed targets out of the top 10
Semantic profile expansion	+0.0003	−0.0060	profile tags match half the catalogue
Query rewrite (deterministic)	−0.0029	—	pipeline already normalises the query
Query rewrite (LLM, gated)	−0.0028	—	matched the regex, cost 6,765 tokens
Question steering on overload	−0.0039	−0.0018	split power is not answerability

They ship switchable and off, each carrying its measurement.

We mis-specified our own held-out set and it hid our biggest win. Our synthetic holdout sampled targets differently from the real benchmark on popularity, which made a genuinely large improvement look flat. Finding that meant re-running the whole comparison table.

Configuration silently ate 0.057 points. .env is gitignored, so a fresh clone scores 0.857 instead of 0.914 with every feature falling back to its code default — and nothing warns you. We moved the winning configuration into .env.example and wrote tools/verify_setup.py, which checks the commit, the catalogue hash against the prebuilt vector index, all sixteen score-critical settings, and whether the models actually loaded.

A real API outage tested the fallback for us. The API returned HTTP 429 insufficient_quota for an entire 200-session run. The evaluation completed normally, with valid results and zero tokens reported.

Accomplishments that we're proud of

0.9143 public / 0.8766 held out, against 0.1067 for the provided baseline — and the held-out gap is small enough that we believe the number.
Confidence-sized responses. Treating "how many results to show" as a measured decision rather than a constant is the single idea the score rests on.
95% token reduction from the ambiguity gate, with no score movement — the agent is cheap and works entirely offline.
Every feature carries its measurement. Nothing shipped on intuition; the rejected list is as documented as the accepted one.
A guardrail that fixed a real fairness bug — the catalogue's gender skew was quietly demoting correct results for half the queries.

What we learned

Benchmarks reward things products don't. Part of our score reflects the metric, not ranking quality. Remove depth capping entirely and we score 0.8585 with MRR 0.662 — that's the honest measure of the ranking. And shoppers would find their item faster without it (MTTC 2.76 vs 3.54). In a real product you always show ten results. We report both numbers because the difference is the interesting part.

Gate the model, don't lean on it. The measured contribution of the LLM path is +0.0002 public, +0.0038 held out. It's enabled because it's cheap and slightly positive, not because we depend on it. Most of the work is deterministic, inspectable, and fast.

Provenance beats content. Typing a constraint by how it arrived — volunteered vs. answered — turned out to matter more than what the constraint said.

Calibrating on the test set is a real limitation. Our separation thresholds come from 70 public sessions. Held-out validation passed, but we should say plainly that the calibration data was the test data. Likewise, our popularity weighting is a bet on how the hidden set is sampled: public targets have a median of 6,846 ratings against the catalogue's 12, worth +0.0147 to us — but −0.0031 if the hidden set is drawn uniformly instead.

What's next for ShopSage

Show all ten, always. Drop depth capping and optimise for the shopper rather than the metric — then rebuild the confidence signal as ordering confidence instead of disclosure confidence.
Earn the cross-encoder's place. Disabling it today measures 0.9150 against 0.9145; it's the semantic reranking stage the brief asks for but it isn't carrying the score. A domain-tuned reranker over apparel attributes should.
Recalibrate on genuinely unseen sessions, with a synthetic set sampled to match the benchmark's popularity distribution — the fix we already know we need.
Handle the unwinnable sessions. public_0144's target has no material or closure metadata and the shopper only ever discloses attributes true of its entire pool. Catalogue-gap detection would let the agent ask a question that actually separates.
Persist the distilled profile across sessions, so preferences learned in one conversation shorten the next one — the distillation already runs, it just doesn't outlive the session yet.