BAD-SMILE-shopping-copilot

Welcome page
Try intend_override on the evaluator
Conversation and recommendation (10 matches)

Inspiration

The brief reads like an LLM problem: understand messy shopper language, track intent across ten turns, rank 50,000 products. We started by building a rule-based baseline purely as a control — something to measure the "real" system against.

The control kept winning. So we stopped building and started measuring, and the project became a different question: how much of this problem actually needs a model, and can we prove it either way?

What it does

A shopper types what they want. Over up to ten turns the agent accumulates their constraints, asks one focused clarification per turn, and returns the ten catalog products most likely to be the one they mean.

On the 200 public sessions it reaches Hit Rate@10 0.995, MRR 0.771, MTTC 1.85, TechnicalScore 0.9118 — against the provided BM25 baseline's 0.107. It finds the target in 1.85 turns on average, and 64.5% of the time the target is ranked first.

It uses no model, no API, no network, and no third-party packages. Reported token usage is 0 and the running cost is $0.

How we built it

Recall merges three SQLite FTS5 BM25 routes: the weighted query, a category-only query, and a strict all-category-terms query that opens only when the dialogue stalls. Rerank scores that pool on six local features — BM25, IDF coverage, phrase containment, category overlap, a popularity prior, budget fit.

The query is built from the entire conversation, not the latest message. A per-session state tracker separates hard requirements from soft preferences and handles intent override by de-weighting superseded constraints instead of deleting them — deleting measured distinctly worse (override hit rate 0.767 versus 1.000).

Every turn also returns a plain-language reason for the top result, and when the top ten score too closely to be meaningfully ordered, the agent says it cannot tell them apart rather than presenting an arbitrary ranking. That admission is calibrated: when it fires, it is genuinely wrong 46.8% of the time; when it does not, it is right over 90% of the time.

The agent re-routes itself at runtime

Four decisions are made per turn from observable session state, so the strategy changes as the conversation does:

Signal	What changes
The turn added no information	A third recall route opens, requiring every category term
Top-10 scores are nearly tied	The reply switches from a ranking to a structured clarification
The same products keep resurfacing	A novelty penalty rotates the list
An attribute is answered or declined	It is retired from the question queue

The conditions matter more than the mechanisms. Running the extra recall route on every turn costs 0.013 — it dilutes rankings that were already right. Gated on "the dialogue stalled" it gains 0.0034, because by then the current Top 10 is demonstrably wrong and there is nothing left to lose.

One thing we deliberately did not build: the brief suggests cutting retrieval off entirely when the candidate pool is overloaded. Under this protocol a turn returning no products cannot convert, so withholding the list spends a turn to buy nothing. The agent returns its ten best and says it cannot separate them — same information, no wasted turn.

Challenges we ran into

We measured the wrong thing, twice. Profile tags looked promising at 1.92x lift, and pseudo-relevance feedback at 2.07x. Both hurt when implemented. Both lifts had been measured against the whole catalog rather than against the candidate pool where ranking actually happens — inside the pool the signal decayed to 1.17x and to nothing. The same trap, twice.

A one-line bug cost us a target. Candidates recalled by a secondary route were scored bm25 = 0.0. That is not "matched poorly" — it is never measured, and it silently applied a full penalty in a space where the entire Top-10 score spread is 0.183. Fixing it recovered a session that no reranking weight could reach.

Our paraphrase robustness problem was not the one we assumed. Rewriting the simulator's phrasing cost ~0.06, which looked like a vocabulary-matching gap and therefore an argument for embeddings. Measuring it showed the opposite: content words reached the query 100% of the time under every rewrite. What broke was slot recognition — category terms survived but stopped being recognised as a category, losing their weight. Widening the opener patterns recovered most of it in a few lines of regex. Embeddings would not have helped.

What we learned

We ran the model paths rather than assuming. A local embedding model discriminated worse than our existing lexical features (1.14 against 1.31); ranking by it alone pushed the target lower in 24 of 38 sessions. A local LLM lost to the rules on every phrasing the rules cover (96.7% against 100%) — but won decisively where the rules fail completely (93.3% against 0%). So the LLM ships as a fallback, consulted only when the rules extract nothing, disabled by default.

The most useful thing we found is a pattern across every experiment:

Every change that improved the score corrected a distortion in a signal already in use. Every change to the retrieval strategy — diversity reranking, hard filtering, per-track weights, pseudo-relevance feedback, dense retrieval, LLM reranking — measured neutral or negative.

An exact-match protocol rewards ordering a candidate set, not reshaping it. Reshaping adds no information; it only perturbs a ranking that is already tuned. Eleven hypotheses were rejected on measurement, each with a split-half check because we twice caught ourselves fitting noise.

What's next

The remaining miss has a target with a single customer review — the popularity prior buries it, and weakening that prior costs more sessions than it saves. That is an information limit, not an engineering one.

The genuine ceiling is the parser: it is the one component whose failure modes are not bounded by the protocol. More weight tuning would be fitting noise; we measured the ceiling and it is flat.