Larpers Go Shopping

Inspiration

Anyone who's used a shopping chatbot knows the moment it breaks character. You say you want running shoes, then a few messages later mention you actually need them waterproof for trail runs, and the bot cheerfully suggests the same road-running sneaker it showed you three replies ago. It never forgot what you were shopping for. It forgot that you were talking to it. Every message gets treated like a fresh search box, and whatever nuance you built up gets thrown away the second you hit enter.

That's the part that actually bothered us. A real salesperson doesn't re-ask what you're looking for every time you add a detail, they update their picture of you. Vague turns into specific, "casual" turns into "actually I need this for a wedding," and a decent salesperson quietly drops the earlier assumption instead of holding you to your first sentence forever. We wanted to see how far a shopping agent could get if it treated a conversation as evidence that keeps building, where the product category stays fixed but everything else can sharpen, shift, or get overridden the more you talk, instead of treating every message like it just met you.

What it does

Larpers Go Shopping is a deterministic, LLM-free conversational retrieval and reranking pipeline. Given a multi-turn shopping conversation, it:

• retrieves candidates with field-aware BM25 (title, categories, features, description, store, each weighted separately), backed by a lightweight LSA semantic index trained on the 50k-product catalogue for when lexical overlap isn't enough;
• accumulates conversational evidence across turns, keeping the shopper's product category permanently anchored while individual preferences can be added, superseded, or explicitly overridden;
• reranks deterministically on category coverage, evidence-phrase matches, hard budget constraints, and popularity/rating tie-breaks;
• asks the single most useful clarifying question when candidates are still ambiguous, based on candidate-set entropy instead of a fixed script;
• decides, turn by turn, whether to lead with one best guess or show a full top-10, depending on how much room is left to recover if it's wrong.

How we built it

Three layers do the actual work: retrieval finds candidates, state tracks what the shopper has revealed, and ranking decides what to show.

• Retrieval. The 50k-product catalogue is indexed at startup into an in-memory SQLite FTS5 table, with each metadata field weighted separately in the BM25 query (title 6.0, categories 4.0, features 2.5, details 2.5, store 1.5, description 1.0) so a match in a product's title counts for more than a passing mention in its description. A TF-IDF + SVD index, fitted offline and shipped as a 9MB asset, provides a dense fallback route when lexical overlap alone retrieves too little.
• State. Each turn's message is parsed into evidence chunks carrying a recency weight of 0.8^age, so late specific constraints outweigh early vague ones without ever fully discarding them. The product category is extracted once and anchored permanently, since that's the one thing shoppers don't silently revise. Budget is pulled out as structured numeric metadata through a staged parser (money context → approximate language → range or bound) and applied as a genuine eligibility filter rather than a scoring nudge, with negated and cancelled budgets handled explicitly.
• Ranking. Candidates are scored on category coverage and per-chunk evidence coverage, with exact phrase matches against product metadata weighted highest, then tie-broken on popularity and original BM25 position. When no evidence exists yet, BM25 order takes precedence over raw popularity, since textual relevance beats a review count when there's nothing else to go on.

Challenges we ran into

Overfitting to the public evaluator

We only had 200 labeled sessions to look at, and it's easy to unconsciously tune an agent until it memorizes the shape of those 200 conversations instead of actually getting better at shopping, especially knowing there were 800 more private sessions we'd never get to see or tune against. We got out of that trap by refusing to trust the public number on its own:

• built a second, self-supervised evaluation layer straight out of the product catalogue: take a real attribute like "waterproof," strip the literal keyword out, rephrase it as something like "something designed to keep water from getting through," and check whether the agent can still find the right product without ever seeing the original word;
• ran a full shadow evaluation that replayed the agent's actual production interface against those paraphrased conversations, not just the retrieval step in isolation;
• rejected several changes that looked great on the public 200 sessions because they did nothing, or actively hurt, on these catalogue-derived tests;
• kept a few changes that barely moved the public score at all, because they clearly helped generalization on the robustness tests.

Ideas that looked airtight and weren't

• One tried to make slow-resolving conversations wrap up faster by treating a shopper's "I don't have a preference" answer to the generic catch-all question as a sign the conversation was basically done. On paper that saves a turn. In practice it also flipped the agent into a completely different, wider-search ranking mode before it had gathered any real evidence, and it wrecked accuracy on exactly the sessions it was meant to help.
• Another tried to make the clarifying question smarter early on by picking whichever attribute best split up the current candidate pool, which sounds reasonable but actually optimizes for the wrong thing: it targets whatever makes candidates look different from each other, not whatever the shopper is likely to actually have an opinion on.

Both got built, measured honestly, and thrown out once the numbers said so, which is a harder habit to keep than it sounds.

Performance drops as the evaluation set gets broader

The most sobering finding came late: when we evaluated on larger and more diverse data slices instead of a narrow development subset, both Hit Rate and MRR quietly got worse. That's the concrete confirmation that a saturated public score can mask a real generalization gap rather than prove it away. Rather than treat the perfect public Hit Rate@10 as a finish line, we now treat it as the floor, and closing that gap on broader, less template-aligned evaluation is the single biggest open problem left, not a solved one.

Accomplishments that we're proud of

• Kept the production pipeline fully offline and deterministic, with zero API cost and zero prompt or completion tokens, since official scoring may run with restricted network access and we didn't want that to be a risk. We did try a more powerful neural semantic retriever during development, but it didn't consistently translate into better end-to-end recommendation metrics, so we kept the lightweight local model that actually earned its place.
• Built a whole second evaluation axis alongside the official public evaluator: a self-supervised, catalogue-derived robustness benchmark (concept paraphrase retrieval, end-to-end shadow sessions, budget parser tests, semantic-fusion ablations) so improvements could be trusted beyond the 200 labeled public sessions, ahead of the 800-session private set we never got to see.
• Held a strict "ship only evidence-backed changes" rule across many iterations. Every regression we hit got caught by ablation, traced to a root cause, and either reverted or documented honestly instead of buried.
• Reached a perfect Hit Rate@10 on the official 200-session public evaluator while still being honest that a saturated public score isn't the same as solved generalization, and building the evaluation infrastructure to actually show that gap instead of hiding it.

What we learned

The biggest lesson was that a saturated benchmark is a sign to build a second benchmark, not a sign to stop measuring. Once the public 200 sessions stopped being able to tell good ideas from bad ones, everything we shipped got validated against catalogue-derived tests we built ourselves instead.

We also learned to trust, and actually report, negative results. More than one experiment that sounded good on paper was ablated, measured honestly against the full evaluation suite, and quietly shelved instead of shipped once it turned out to regress real metrics or add complexity without earning it. Preferring measurable improvement over architectural complexity became one of our actual design principles, not just a nice sentiment. That honesty mattered more than any single reranking trick, and it's what let the project keep improving without quietly breaking something we'd already fixed.

What's next for Larpers Go Shopping

• Actually measure the generalization gap instead of assuming it away: run the same agent across increasing evaluation sizes and multiple random seeds (50, 100, 250, 500, 1000+ sessions) to tell real improvements apart from ones that just happen to fit a convenient subset.
• Broaden query planning beyond our one hand-built synonym cluster, mapping many more natural shopper phrasings ("helps reduce heat buildup," "secure traction," "easy to adjust") into canonical catalogue concepts before retrieval, without letting that expansion take control of final ranking.
• Add a small local semantic model as a conditional fallback rather than an always-on component, triggered only when retrieval confidence is low, so ordinary queries stay fast while harder paraphrases get a stronger recovery path.
• Make clarification genuinely information-theoretic: estimate how many candidates each possible answer would actually eliminate and ask whichever question has the highest expected information gain, instead of relying on coverage and entropy as a proxy.

Every future change will keep being chosen on measured generalization, not on how well it fits the same 200 public sessions we've already seen.

Built With

• numpy
• python
• pytorch
• regex
• scikit-learn
• sentence-transformers
• tf-idf
• unittest