Shopping copilot

Inspiration

TechJam's Problem Statement 4 posed a deceptively hard search problem: a customer has one specific product in mind out of 50,000 items, and all we get is a short opening message and up to ten conversational turns to find it. Most "AI shopping assistant" demos lean on an LLM to sound helpful. We wanted to know something narrower and more honest: how much of "helpful" is actually retrieval and dialog-state engineering, and how much genuinely needs a language model? That question — chase the deterministic pipeline as far as it goes before reaching for an API key — became the whole project.

What it does

Each turn, the agent parses the customer's message for known attributes (material, color, size, brand, budget...), updates a running ledger of what it knows, retrieves candidates with BM25 + a semantic reranker, and either returns a ranked top-10 or asks oneengineering, and how much genuinely needs a language model? That question — chase the deterministic pipeline as far as it goes before reaching for an API key — became the whole project.

What it does

Each turn, the agent parses the customer's message for known attributes (material, color, size, brand, budget...), updates a running ledger of what it knows, retrieves candidates with BM25 + asemantic reranker, and either returns a ranked top-10 or asks one targeted clarifying question chosen to split the current candidate pool as evenly as possible. It has to recover gracefully when acustomer reverses themselves mid-conversation ("actually, forgetwaterproof — just show me something cheap") without losing the parts of their intent that are still valid.

How we built it

The pipeline is six stages, all deterministic:

$$ \text{Parse} \to \text{Update state} \to \text{Classify route} \to \text{Retrieve (BM25)} \to \text{Rerank} \to \text{Clarify or recommend} $$

• State — a ConstraintLedger tracks every constraint the customer has volunteered or answered, with its arrival turn, so an "actually, never mind" can revoke just the stale slots and keep the rest. This one change took our HitRate@10 from 0.160 to 0.870 in a single experiment.
• Retrieval — SQLite FTS5 (BM25) over the accumulated constraintterms, built in-memory at startup.
• Ranking — a weighted sum of signal:

$$ \text{score} = w_{\text{fields}} \cdot \text{match} + 1.2\ln(1+\text{rating_count}) + 2.0 \cdot \mathbb{1}[\text{has_price}] + \cos(\text{TF-IDF+SVD}) + \text{phrase bonus} + 4.0 \cdot \mathbb{1}[\text{all constraints matched}] $$

• Clarification — the agent asks about whichever attribute is best-covered and most varied among the current top-100 candidates, so the answer is maximally informative rather than generic.

Everything runs offline, with zero LLM tokens and $0.00 marginal cost per session, on plain scikit-learn TF-IDF/SVD for the semantic term.

Final numbers on the 200 public sessions, unmodified evaluator:

Challenges we ran into

• The field-weighting bug that cost us the most, for the longest. For 28 experiments we weighted product titles above category names when scoring matches. It turns out the evaluator writes the customer's opening message verbatim from the target's own category path — socategories is by far the most reliable field to match on, and we had it under-weighted the whole time. Fixing that one number (categories: 6.0) was our single largest late-stage gain.
• Two "obviously right" ideas that measurably hurt. The brief suggested routing browsing sessions to dense retrieval and using intent classification to steer per-route weights. We built both, measured them properly on a held-out validation split, and both made things worse — dense retrieval alone scored 0.670 vs. BM25's 0.995, and even hybridizing it cost accuracy by evicting correct BM25candidates from a truncated pool. Reporting why something we tried didn't work felt as important as reporting what did.
• Deciding not to use an LLM. We measured the actual ceiling:disabling the semantic term entirely only cost us 0.0019 of TechnicalScore. That's the entire prize available to any smarter query rewriting — not enough to justify the latency, cost, andnon-determinism of an LLM call, especially with scoring possibly running network-disabled.
• Finding failure modes the official metrics couldn't see. We built our own stress tests — masking catalog fields, paraphrasing the customer's wording — and found the system's biggest real weakness: if you strip the category phrase out of the customer's message entirely, TechnicalScore drops from 0.917 to 0.767. Ordinary paraphrasing barely matters; naming the category matters enormously.

What we learned

Rigor mattered more than cleverness. Separating a validation split from the full evaluation set caught at least one change that looked like a win on partial data and reversed on the full set. Keeping a log of every rejected experiment — 26 of 46 were rejected — made it possible to trust the numbers that survived, instead of just the oneswe liked. And the biggest lesson: a well-tuned, fully deterministic, zero-token pipeline can outperform intuition about what "should" need an LLM, if you're willing to actually measure instead of assume.

Built With

• bm25
• conversational-ai
• natural-language-processing
• numpy
• python
• react
• recommender-system
• scikit-learn
• tf-idf
• typescript
• vite