Shopping Copilot

Inspiration

Search boxes make shoppers translate what they want into keywords. We wanted to test a different idea: people often describe products by quoting them. Someone doesn't say "date display," they say "Day / Date Indicator," because that's the phrase printed on the product page. We figured a search pipeline built around that habit, instead of one that dissolves everything into a bag of words, could beat standard hybrid retrieval by a lot. TikTok TechJam's Shopping Copilot problem gave us a real 50,000-item catalog and a local evaluator to actually test that against, so we took it.

What it does

Given an anonymized shopper profile and a short opening message, the agent works through a 50,000-product Amazon catalog and tries to land on the right item in as few turns as possible. It only asks a clarifying question when it's genuinely unsure which of a few likely candidates the shopper means, and it still returns a best-effort Top 10 in that same turn. It's built to handle four kinds of sessions: open-ended browsing, precise buying, a shopper changing their mind mid-conversation ("actually, it has to be leather"), and edge cases at the boundary of what the catalog covers.

On the organizer's 200 public dev sessions:

	Hit Rate@10	MRR	MTTC	TechnicalScore
Weak BM25 baseline	0.125	0.068	9.81	0.107
Our agent	0.995	0.778	1.73	0.916

How we built it

The pipeline runs in this order: parse the turn, update the dialog state, route it, run multi-route retrieval, fuse the results, rerank, then decide whether to clarify. Everything runs in memory and doesn't require any network calls.

Three retrieval routes run on every turn and get combined with reciprocal rank fusion. There's a BM25 keyword route using SQLite FTS5 with Porter stemming, a dense TF-IDF+SVD similarity route diversified with MMR for browsing, and a verbatim attribute-phrase index. That last one came from noticing that 89.6% of the catalog's 220,350 indexed attribute phrases show up on exactly one product. If a shopper's wording matches an attribute phrase closely, that's often enough to identify the item on its own.

Routing isn't a binary switch between buying and browsing. We compute a continuous buying weight each turn, based on the message and on how many requirements have actually been confirmed so far. A session that opens vague and firms up by turn 3 shifts toward precision retrieval gradually, without ever needing to be told which mode it's in.

The dialog state machine accumulates requirements across turns, and when a shopper overrides an earlier choice, the old value gets demoted to a weak prior instead of thrown away. Untouched slots decay over time, and a settled "no preference" is treated differently from a one-off "you decide," which gets asked again later instead of being burned for good.

For clarification, each askable attribute gets scored on how much it would narrow the current candidate pool, discounted by how likely this particular shopper is to actually have an answer, which we estimate from what earlier questions in the session returned.

The final ranking stage is entirely local and model-free. It blends requirement coverage, how prominently a matched requirement sits in the item's own attribute list, a demand prior from review volume, price agreement, the fused retrieval rank, and the shopper's long-term profile.

Tools, libraries, APIs, and data

Development tools: VS Code, Git, Python 3.14 (3.10+ supported), unittest for testing.

APIs: none required to hit our numbers. The default pipeline runs fully local and deterministic, with zero network calls at runtime. We built one optional accelerator: an LLM re-rank of the final shortlist using claude-haiku-4-5, gated behind an API key and off by default, so the submission never depends on a live credential during grading.

Libraries and frameworks: numpy and scikit-learn (TfidfVectorizer, TruncatedSVD) for the dense retrieval route, and Python's stdlib sqlite3 with the FTS5 extension for BM25 keyword search. Optionally, sentence-transformers as a swappable embedding upgrade.

Datasets and assets: the organizer's frozen 50,000-product catalog and 200 labelled public dev sessions, drawn from Amazon Reviews 2023's Clothing_Shoes_and_Jewelry category (McAuley Lab, UCSD). The catalog was treated as strictly read-only throughout, no external or hand-labelled data was added, and no model was trained or fine-tuned.

Challenges we ran into

Balancing precision against recall in the reranker was harder than expected. Early on, we weighted broad attribute matches like color or material too heavily, which got the right item somewhere in the Top 10 but rarely at rank 1, and that specifically hurt MRR. We had to pull out a sharper signal for "this is the exact item" using attribute-phrase and title matching, and keep it separate from looser category-level agreement.

Slot decay was trickier than it looked. Forgetting stale constraints sounds like an obvious good idea, but tuning it too aggressively meant the agent forgot real requirements right around the turn it needed them most. We only caught this by looking at scenario-level metrics instead of the aggregate score, which had been masking it.

We also didn't have ground truth beyond the 200 public sessions. Every tuning decision had to be validated on that small dev set rather than the 800 private sessions we're actually judged on, so we treated our public numbers as a signal to guide us, not a target to chase directly.

Accomplishments that we're proud of

We reached 0.995 Hit Rate@10 and 0.778 MRR with a pipeline that's entirely local, deterministic, and free to run, no LLM calls needed to hit those numbers. Intent Override sessions, which we expected to be the hardest, actually score the highest MRR of any scenario at 0.942. The state machine handles a shopper changing their mind better than it handles a first-shot request.

What we learned

A lot of what people reach for an LLM to solve turns out to be recoverable from cheaper signals if you look closely at how people actually phrase things in a specific domain. Verbatim quoting ended up being a stronger and cheaper signal here than generic embedding similarity. We also learned that a single aggregate score can hide exactly which scenario a change is helping or hurting, so per-scenario breakdowns became something we checked before trusting any tuning decision.

What's next for Shopping Copilot

We'd replace our hand-built attribute vocabulary with one derived automatically from the catalog's own details fields, so it generalizes past clothing and jewelry. We'd add an LLM query-rewriting step ahead of retrieval for heavily paraphrased requests, which our verbatim-matching route currently misses. And we'd fit our ranking weights on held-out data instead of by hand.