Shopilot — AI Conversational Shopping Agent

The problem, in the baseline's own words

Ask the official BM25 starter for "something for a winter wedding" and it hands back "Photographer Gift Create **Something* Beautiful"*.

It matched on the word something.

That isn't a bad implementation. It's the ceiling of the approach. Nothing in the catalog is called "winter wedding", so the words in the query and the words in the catalog never meet, and you end up needing to know the answer before you can search for it.

Shopilot scores 0.954 where that baseline scores 0.107.

What it does

It's a multi-turn shopping agent over a frozen 50,000-product Amazon catalog, and every turn it makes exactly one decision: do I know enough to answer you, or should I ask you one more question?

Scored by the organizer's own evaluator, unmodified, across all 200 public sessions:

Every session solved, median rank 1, 2.8 turns against a budget of 10. It spends zero tokens doing it. No model API, no network, no dependencies — Python 3.9's standard library and nothing else.

How we built it

The brief sets out four pillars, and each one lands in its own module.

Routing and retrieval. src/route.py scores a continuous specificity, which splits a precise buying filter from a diverse browsing spread. They're two ends of one dial rather than a branch, so the agent never commits early on thin evidence. src/rank.py then fuses three independent routes: exact constraint matching, BM25, and TF-IDF cosine, each normalised to its own maximum before they're weighted per track.

Browsing also runs a dense recall arm across the whole catalog and reserves its last two slots for candidates outside the named category. That last bit is what turns "cross-category matching" from a claim into a behaviour. We checked before and after: previously the top ten were 100% in-category on all 80 browsing sessions, so the capability the brief asks for simply didn't exist. It's now 20.9%, at a measured composite cost of 0.000.

Dialog state. src/state.py accumulates weighted constraints and decays them over time. On an intent override it demotes rather than erases, which is a deliberate departure from the brief: the abandoned preference is still a true attribute of the hidden target, and erasing it costs 0.009 when you measure.

Choosing the question. This is the part that carries the score. Rather than working down a list, the agent recomputes the expected information gain of every attribute over the live candidate pool each turn and asks whichever one splits it most sharply. Switch it off and the score falls by 0.379, more than every other component put together.

Evaluation. 261 tests, an 18-arm ablation, a paraphrase stress test at three levels, and 5-fold cross-validation. make docs re-derives every number in the README from a committed artifact and fails the build when the prose drifts.

Challenges we ran into

The metric and the brief disagreed twice, and both times we sided with the measurement and documented it. The brief asks for slot erasure on intent override; erasing costs 0.009 against the real simulator, so we demote. The brief asks browsing for cross-category reach; the evaluator always plants the target inside the named category, so reaching outside it can only displace correct answers. We built it anyway, in the tail of the list where it costs nothing, and said as much.

Then there was the awkward discovery that almost everything we tried landed inside the noise. So we went and measured the noise. Five-fold cross-validation puts this benchmark's fold-to-fold standard deviation at ±0.0116 at n=200. Four of five folds preferred a different late_turn constant to the one we ship, worth +0.0014 — about a fifth of the noise. We didn't take it. That constant governs how deep the ranking gets searched before the turn budget runs out, and tuning to the fifth decimal of 200 sessions is exactly how you win a public set and lose the private one.

The last one was making it usable by an actual person without contaminating the scored path. The evaluator speaks four fixed frames; a human says "a gift for my wife". So there's an assist layer that resolves occasions to real categories, answers greetings without spending a turn, and shows every rewrite it makes. All of it sits behind a flag the evaluator never sets, with tests asserting the four frames pass through untouched.

What we learned

Prose that quotes a measurement is code that can be wrong. Our README claimed a filter caught "279 of 1,115 categories". It was 189. We'd summed two overlapping sets instead of taking their union, and nothing caught it because a human had counted it by hand. make docs now recounts every figure like that from the catalog and fails the build on drift.

The other lesson was about honesty in the results table. Ours says plainly that BM25, the vector route, MMR and personalisation don't earn their keep on the public set. They stay because they're insurance against paraphrase, and we measured that too: 0.815 under light rewording against 0.954 verbatim. Keeping a component you can't show helping is defensible. Quietly implying it helps isn't.

What's next

A live A/B of the optional LLM reranking stage. It's built and tested against a stub, but it ships off, because every number here comes from the offline configuration and we're not going to claim a result we haven't run.

Built With

• amazon-reviews-2023
• bm25
• conversational-ai
• cosine-similarity
• css3
• html5
• http.server
• information-gain
• information-retrieval
• information-theory
• javascript
• mmr
• python
• recommender-systems
• standard-library
• tf-idf
• unittest
• zero-dependency