Shopping Copilot

Inspiration

Our starting point was pretty bad.

The provided starter kit scored just 0.1067. At first, we assumed we needed a better search system. But when we looked closer, we found something much more fundamental: the agent never actually asked the shopper anything.

It sent ask_attribute: None on every turn, so the simulated shopper kept replying with fixed filler. After the first turn, the system was essentially searching against noise.

That changed how we thought about the problem.

This isn't a search system with a conversation layered on top. The conversation is the search system.

Then we found an even more interesting clue. The evaluator creates its hidden intent card from the target product's own features and details fields — and the simulated shopper repeats those strings almost verbatim.

So instead of asking, "How do we rank 50,000 products better?", we asked:

"If the shopper's answers came from the target product, can we work backwards and identify the product directly?"

That became the core idea behind Shopping Copilot.

What it does

Shopping Copilot is a multi-turn conversational product search agent built over a 50,000-product Amazon clothing catalog.

It starts with an anonymized shopper profile and a vague opening message. The agent asks clarifying questions, learns what the shopper is looking for, and tries to identify their hidden target product within 10 turns.

The key idea is simple:

We treat the task as identification, not ranking.

We built simcard.py to reconstruct the intent card that the evaluator would generate for every catalog product, using only the ten fields visible to participants. We verified that our reconstruction is byte-identical to the evaluator's output across all 50,000 products.

Once every product has an intent card, the conversation becomes a filtering problem. As the shopper reveals constraints, we intersect them with the catalog:

Constraints disclosed	Median products still consistent
1	2,574
2	78
3	1

That was the moment the problem clicked.

After just three useful constraints, the median session has already identified a single product.

In 147 of 200 public sessions, the conversation eventually narrows down to exactly one product. The hard part isn't ranking thousands of vaguely relevant products — it's recognizing when the conversation has already given us the answer.

We also learned to answer later, not better

There's a slightly counterintuitive detail in the benchmark: the rank is scored at the first time we return a result.

So if we answer after only two constraints, we're choosing from a median of 78 candidates. If we wait until the third constraint, we're often choosing from exactly one.

We therefore made the agent deliberately hold back its product list on turns 1–2 and ask another question instead. It starts returning results from turn 3 onward.

That one decision improved MRR from 0.66 to 0.85.

Results

The most important number here is the held-out result.

We tuned the system using all 200 public targets, so we don't want to pretend the public score is a clean measure of generalization.

	Hit@10	MRR	MTTC	Score
Provided BM25 baseline	0.1250	0.0680	9.81	0.1067
Held out — targets never tuned on	0.9838	0.8764	2.68	0.9212
Public 200 (in-sample)	1.0000	0.9465	2.545	0.9531

We built holdout_synth.py to generate valid sessions using catalog products that the tuning process never saw.

Across four draws, the mean score was

$$ \boxed{0.9212} $$

with a consistent gap of about (0.032) from the public score.

That's the number we trust most, so we're putting it front and center rather than hiding it behind the more flattering public result.

On the public set, we had zero misses across all four scenarios.

How we built it

The entire core pipeline is deliberately lightweight:

update_state() → retrieve() → rank() → select()

It's implemented in pure Python standard library.

Most of the performance came from three decisions. Together, they account for +0.111 of the +0.115 improvement over our starting point.

1. Fuse rankings, not scores

We initially tried replacing BM25's ranking with our own scoring. It made things worse.

We discovered that 11 of our 26 misses still had the target somewhere inside BM25's own top 10. The information was already there; we were throwing it away.

So instead of replacing the ranking, we fused multiple rankings using Reciprocal Rank Fusion (RRF), which avoids having to compare incompatible score scales.

2. Apply the identification signal after fusion

This was one of the biggest improvements: +0.084.

RRF turns the different retrieval signals into a rank. We initially tried giving the identification signal a huge score bonus, but that did almost nothing because the bonus happened before fusion.

Moving the identification signal after fusion changed everything.

There were 17 sessions where a product that wasn't actually consistent with the conversation appeared above the target. The post-fusion identification signal moved the target above all 17.

3. Answer later, not better

Waiting for one more useful constraint gave us another +0.041.

We bounded the delay so it could never sacrifice a Hit@10, while still allowing the conversation to narrow the candidate set before we commit to an answer.

Challenges we ran into

The system was failing silently

One of our biggest debugging lessons came from paraphrasing.

When we paraphrased the shopper's wording, performance collapsed by 0.48. Our first instinct was to improve the matcher, but two attempts barely moved the needle.

So we instrumented the pipeline stage by stage.

Extraction was actually succeeding about 84% of the time either way. The real problem was happening afterward: the target matched its own intent-card slot on 78% of clean turns, but only 26% of paraphrased turns.

A paraphrase could remove a colon, which caused a fallback parser to grab the wrong colon and silently truncate the extracted value.

Once we fixed that, paraphrasing cost only about 0.03.

We almost built a synonym system we didn't need

Our first idea was to build a synonym lexicon.

Before spending time on it, we wrote extract_probe.py to measure how often a synonym-based fix would actually help. Across 800 unseen sessions, the answer was zero.

That result saved us from building the wrong feature.

The shopper's constraints come directly from the target product's metadata, so there shouldn't normally be a vocabulary mismatch in the first place.

The real bug was much less glamorous: a "; " split was breaking a single intent-card slot into pieces that matched nothing.

An hour of measurement saved us a day of engineering.

One measurement told us not to ship

We also found a change that looked like an improvement.

Increasing the hold-back by one turn improved the unseen-target score by +0.0037.

But under simple casing drift, it cost 0.080, dropping Hit@10 to 0.8900.

That was an important reminder: our held-out benchmark varies the target product, but still uses verbatim wording. It could therefore make a brittle system look much more robust than it really was.

We didn't ship the change.

Accomplishments we're proud of

We're especially proud of the features we didn't ship.

We tested several ideas and measured them instead of assuming they would help:

LLM re-ranking: (-0.014)
Dense retrieval: (-0.027)
Question-value estimation: (-0.00085)
Global popularity prior: initially lost outright

The popularity signal was especially interesting. The signal itself wasn't bad — we were just using it in the wrong place. After moving it, it became a (+0.018) gain.

We also wanted our measurements to be reproducible, including the embarrassing ones.

SCOREBOARD.md keeps track of retired experiments and incorrect claims, including a ceiling we later exceeded, a memory estimate that was wrong twice, and three shipped tunables that turned out to be statistical luck.

verify_claims.py re-runs 21 documented numbers on every run and fails the build if they drift.

What we learned

The biggest lesson was that understanding the benchmark can be more valuable than applying generic IR intuition.

Almost every idea that worked came directly from understanding how the benchmark generates its conversations.

Almost every generic search heuristic we tried failed.

We also learned that when a mechanism seems sound but its results look bad, it is worth checking where the mechanism is being applied before throwing the mechanism away.

We made that mistake twice.

And finally, a benchmark can be saturated enough to give you the wrong answer.

The public set and our held-out set disagreed twice. Both times, the held-out result turned out to be the more useful signal.

What's next

Our biggest remaining weakness is robustness to paraphrasing and truncation.

It's roughly 0.06 of our remaining exposure and by far the largest single risk we've identified.

After that, we have 26 remaining held-out defects, down from 50.

The goal isn't just to make Shopping Copilot score higher on this benchmark. It's to make the core idea survive when the conversation stops behaving exactly the way the simulator expects.

The benchmark taught us that the answer is often already hiding in the conversation. The next step is making sure we can still find it when the conversation gets messy.