Shopping Copilot: Conversational Product Search

Your new friendly Ai Bot

Inspiration

We started by reading the evaluation transcripts instead of the spec, and one thing jumped out. Shoppers say the same thing two completely different ways.

  LITERAL                          NATURAL
  "Buckle closure"                 "a belt that buckles"
  "100% Leather"                   "made of real leather"
  "Water Resistant"                "something that keeps water out"
         |                                    |
         v                                    v
  exists word-for-word              exists nowhere in the
  in the product listing            catalog as a string
         |                                    |
  a lookup problem                   a meaning problem

Those need opposite machinery. Anything fuzzy makes the left column worse. Anything literal fails the right column outright.

We wanted to be good at both, in the same conversation.

What it is

It scores 0.972125 on the official 200 session evaluation using zero API calls. No model, no network, no credential. Clone it, run the evaluator, and it works with nothing configured.

It is also equipped with an LLM and dense semantic retrieval, so language that matches no template still gets understood rather than dropped. "Buckle closure" and "a belt that buckles" are both handled, by different machinery, inside the same conversation.

Both modes are fully supported. The agent checks what the environment provides at startup and runs those components. With a credential the semantic layer engages. Without one it goes quiet, every reply still comes back, every session still completes, and the score still lands.

	Needs install	Needs embedding cache	Needs API key	Cost
Constraint retriever	No	No	No	Free
Semantic retrieval routes	Yes	Yes	No	Free
LLM call sites	Yes	Yes	Yes	~$0.02/session

The scored run is the first row: pure Python standard library. No pip install, no virtualenv, no model download, no 890 MB cache, no key. Two commands and it reproduces 0.972125.

Neither the embedding cache nor a credential arrives with a git clone, so the default state of anyone who downloads the repository is the zero cost, zero network configuration. The semantic layer is opt in, and turning it on changes nothing about that baseline run.

How it works

                        shopper message
                              |
                              v
                    +-------------------+
                    |   intent router   |   BUYING or BROWSING
                    +-------------------+   reweights every route
                              |
              +---------------+---------------+
              |                               |
              v                               v
   +---------------------+        +------------------------+
   | CONSTRAINT RETRIEVER|        |   SEMANTIC RETRIEVER   |
   |                     |        |                        |
   | exact phrase index  |        |  lexical               |
   | SQLite FTS5         |        |  dense: product        |
   | BM25                |        |  dense: features       |
   | category / price /  |        |  dense: description    |
   | popularity priors   |        |  dense: details        |
   |                     |        |  category              |
   | no model            |        |  brand                 |
   | no network          |        |  structured details    |
   | zero tokens         |        |  IDF exact + slot      |
   +---------------------+        +------------------------+
              |                               |
              |     confidence gate           |
              +---------------+---------------+
                              |
                    Reciprocal Rank Fusion
                              |
                              v
                    ranked recommendations

The constraint retriever runs every turn and reports how much of the shopper's stated requirements it matched exactly. High coverage means its ranking stands. Low coverage means the semantic routes run, with the constraint retriever's full ranking fused in as one more input.

Results

Official evaluator, 200 session public set, unmodified:

Metric	Score
TechnicalScore	0.972125
Hit Rate@10	1.000
MRR	0.978083
Mean Turns to Conversion	2.065

Correct product found in every single session, across all four scenarios:

Scenario	Sessions	Hit@10	MRR	MTTC
buying	80	1.000	0.9838	1.52
browsing	80	1.000	0.9698	1.98
intent_override	30	1.000	0.9778	3.63
boundary	10	1.000	1.0000	2.40

Here is a real session, including the moment the shopper changes their mind:

TURN 1
  shopper >  I'm looking for Accessories Belts. Buckle closure
  agent   >  Got it. Anything else that matters, fit, colour, or budget?
     path: constraint    latency: 19 ms    tokens: 0
        1. Hide & Drink, Rustic Handmade Full Grain Leather Men  <<< TARGET

TURN 3
  shopper >  Actually, ignore my earlier preference. What I need is: leather.
  agent   >  Got it. Anything else that matters, fit, colour, or budget?
     path: constraint    latency: 56 ms    tokens: 0
        1. Hide & Drink, Rustic Handmade Full Grain Leather Men  <<< TARGET

  >>> FOUND IT at rank 1 on turn 3

The belief state decayed the replaced preference without discarding the rest of the conversation, so the target held rank 1 straight through the change of mind.

Why this matters outside the benchmark

Conversational search is easy to prototype and hard to run. The reason is almost always cost and reliability, not model quality.

Cost. A system that calls a model on every turn pays on every turn. At roughly $0.02 per session, a million shopper sessions is about $20,000 a month, and that scales linearly forever. Our default path costs nothing per session because it never calls a model. The LLM engages only on turns the deterministic retriever cannot resolve confidently, so spend tracks difficulty instead of traffic.

Latency. Measured per turn:

  constraint retriever      19 to 56 ms
  LLM call sites            2000 to 6000 ms

Two orders of magnitude. In retail, latency is conversion. A system that answers most turns in tens of milliseconds and reserves seconds for genuinely hard ones is the difference between a feature that ships and one that gets cut in review.

Reliability. Our API credits ran out in the middle of a 200 session evaluation. The agent did not crash, did not return empty responses, and finished the run at full score. That is not a hypothetical: an e-commerce search box that goes down when a vendor has an outage is not deployable, and we have watched ours survive exactly that.

Portability. Nothing here is specific to clothing. The constraint index is built from whatever attributes a catalog already publishes, so the same approach transfers to any domain with structured product text: electronics specs, grocery labels, parts catalogs, property listings.

Feasibility

We deliberately kept this buildable rather than impressive on paper.

Constraint	How it is met
No external infrastructure	Runs entirely in memory, no vector database, no cluster
Runs on modest hardware	Whole system runs in one process on a laptop
Startup cost	~30 seconds to index 50,000 products
Dependencies for the scored path	Zero. Python standard library only
Cost is bounded, not open ended	LLM calls are gated, capped, and optional
Fails safe	Every failure mode degrades to a working configuration

The heaviest component, the 890 MB embedding cache, is also the most optional one, and a smaller distilled encoder would cut it roughly in half without touching the architecture.

Challenges we ran into

Reciprocal Rank Fusion has a voting problem nobody warns you about.

RRF puts every route on the same scale no matter how confident that route is. Correct when all routes are guessing. Badly wrong when one route is certain.

  product A:  matched the shopper's exact words
              weak on every fuzzy route
              -> 1 strong vote

  product B:  mediocre on every route, strong on none
              -> 5 weak votes

  RRF verdict: product B wins

Five weak votes outvoted one certain one. Literal match evidence now contributes both a rank and a direct score, weighted by inverse document frequency, so a rare phrase counts heavily and boilerplate does not.

BM25 punished a product for being well documented.

That same belt ranked 57th of 500 on lexical search despite containing "Buckle closure" character for character. The cause was document length: it carried five long marketing paragraphs including a "101 Year Warranty" blurb, which diluted term density enough to bury a perfect match.

One weight was wrong by two orders of magnitude.

Weight	Before	After	Gain
Popularity	0.02	2.2	+0.0100
Category	13.0	30.0	+0.0014
Exact match	6.0	3.0	+0.0013

Popularity alone was three quarters of the total gain. The evaluation targets are real purchase records, and purchases concentrate on products people actually buy, so review volume is genuine evidence rather than a tie breaker.

Our API credits ran out mid evaluation.

A 200 session run lost its LLM access partway through. The agent did not crash, did not return garbage, and completed on its deterministic path. That turned an abstract claim about graceful degradation into something we had actually watched happen.

The embedding cache is 890 MB and GitHub rejects files over 100 MB.

One file alone is 727 MB. We ship a build script that regenerates it in about 25 minutes, and publish the prebuilt cache as a GitHub Release, which accepts assets up to 2 GB.

What we learned

We assumed the domain matched encoder would win. It lost.

BLaIR was pretrained on Amazon Reviews 2023, the same data family as this catalog. It knows the vocabulary. Obvious choice.

Encoder	200 sessions	20 session subset	Stable?
BLaIR (domain matched)	0.9011	0.8585	swings 0.04
BGE (general retrieval)	0.9068	0.9000	swings 0.007

BLaIR was trained to associate items with review text, not to match short queries against passages. Its similarity scores on this catalog sat in a narrow band, median around 0.78 across 50,000 products, leaving very little room to separate a real match from a random one. BGE is trained contrastively to push those apart.

Better retrieval beat a better language model, and it was not close.

  swapping the encoder      +0.0415
  best LLM reranker         +0.0130   (and it costs money every turn)
  cheap LLM reranker        -0.0405   (worse than no reranker at all)

A weak reranker takes good retrieval and reorders it into worse retrieval. We had been tuning the expensive component while the cheap one held the bottleneck.

Measure the thing you are about to argue about.

Almost every belief we held at the start was testable in under a minute, and several were wrong. The single biggest gain in the project came from a weight we had confidently set two orders of magnitude too low.

What's next

A paraphrase harness that rewrites the requirement itself rather than the sentence around it. Our current harnesses reword the frame but pass phrases like "Buckle closure" through unchanged, which under tests exactly the case the semantic retriever exists for.

Beyond that, moving requirement order tracking off the regex parser and onto the language model so a reworded change of mind keeps its alignment, and distilling the encoder so the semantic path ships in megabytes instead of hundreds of them.