Banner

Shopping Copilot: AI Conversational Search and Recommendations

Inspiration

Every e-commerce search box asks the same silent question: type the right keywords, or get nothing useful. Real shoppers don't think in keywords. They think in half-formed intentions ("something for a wedding, not too formal"), constraints that shift mid-conversation ("actually, forget black, I want navy"), and preferences they haven't even said out loud yet.

We wanted to build something closer to a good salesperson: figure out early whether someone's browsing or buying, ask the right clarifying question instead of dumping ten thousand results, and actually remember what was said three turns ago. The TechJam Shopping Copilot brief gave us a concrete way to chase that idea instead of just talking about it, with four defined pillars and hard metrics to check our work against.

What it does

Given a real Amazon catalog (50,000 clothing, shoes, and jewelry products) and a simulated customer with a hidden target item in mind, our agent has a natural back and forth conversation, up to 10 turns, to find that exact product. Each turn it:

Classifies intent. Is this customer clearly buying ("I need black, size 8") or still browsing ("just looking")?
Searches two ways at once. A precision track over confirmed hard constraints, and a diversity track over everything discussed so far, fused together with reciprocal rank fusion and weighted by how much it currently knows.
Tracks a running memory of every disclosed attribute (material, color, size, budget, style, use case, and so on), and rewrites that memory gracefully if the customer changes their mind mid-conversation.
Decides whether to ask or answer. Since asking costs nothing (recommendations go out every turn regardless), it learns to keep drawing out more information instead of guessing too early.
Re-ranks the combined candidates by blending the search engine's own relevance signal with how well each one matches everything we've been told, including a price check once a budget is known.

On the organizer's own 200-session public evaluation set, this took our TechnicalScore from a weak 0.107 baseline to 0.764. Roughly 7x. Hit rate climbed from 12% to 88%, and the average number of turns to find the right item dropped from around 10 to under 4.

How we built it

We started from the organizer's participant kit: the frozen 50k-product catalog, 200 labeled public sessions, and the local evaluator. We treated the evaluator as ground truth for every decision rather than guessing at what "good" looked like.

The pipeline runs entirely offline, no paid LLM required:

Retrieval: an in-memory SQLite FTS5 index over title, categories, features, details, store, and description, queried through two parallel routes (precision and diversity) each turn.
Fusion: reciprocal rank fusion combining both routes, weighted by a runtime alpha value that shifts toward precision as more hard constraints pile up. This is the self-evolving piece of the four pillars.
Dialog state machine: a per-session slot tracker (material, color, size, budget, style, use case) that accumulates constraints turn over turn, catches override cues like "actually, ignore my earlier preference" and rewrites the relevant slots, then decides what to ask next.
Reranking: a local heuristic scorer standing in for an LLM ranking stage. It blends the retrieval engine's own relevance score with term overlap against everything disclosed, written so it can be swapped for a real LLM call later without touching the rest of the pipeline.

We iterated mostly by reading the evaluator's own source code closely instead of tuning blind. Every change we shipped got checked against a full rerun of the 200-session evaluation, comparing the actual number rather than trusting intuition.

Challenges we ran into

Reading the evaluator, not just the API contract, was the real unlock. Early on we assumed asking about a specific attribute was always the right move. Digging into how the simulated customer actually replies showed it only discloses new information when asked, and that "other" is a wildcard the contract explicitly allows, revealing any remaining undisclosed detail rather than one tied to a guessed category. Realizing the agent should almost always keep asking, since recommendations go out every turn either way, was worth more than any retrieval tweak we tried.

A silent slot-corruption bug. Our first version classified an entire multi-attribute customer reply, something like "leather; color: black," into a single bucket, silently losing everything but the first-matched category. Multi-turn conversations that mixed material and color and size into one reply were quietly dropping two-thirds of the disclosed information.

Filler text posing as signal. The evaluator's own template phrases, things like "I'm looking for X, but I'm still exploring" and "I don't have an additional preference for other," were getting stored as if they were real product constraints, diluting the genuinely useful signal. Filtering these out specifically pushed the boundary scenario (customers with no strong preference) to a perfect 1.0 hit rate.

Throwing away a good signal while trying to improve it. Our reranker originally discarded the search engine's own relevance ranking and rescored everything from scratch using flat term counts, which ignores how rare or common a word is across the catalog. BM25 already accounts for that; a plain count doesn't. Blending the original ranking back in as a weighted prior, instead of overriding it outright, was one of the biggest single score jumps we made.

Environment friction. Getting a teammate from "the code exists" to "the code runs on my machine" taught us its own lessons. PowerShell aliases curl to a cmdlet that doesn't understand Unix flags, stale pycache folders can mask a fix that was already saved correctly, and a copy-paste can silently land in the wrong file. Small stuff, but it ate real time before we started verifying file contents directly instead of assuming a save had worked.

Accomplishments that we're proud of

Took the TechnicalScore from 0.107 to 0.764, about 7x, purely through architecture and evaluator-aware tuning. No external API, no paid model, fully reproducible offline.
Every improvement we claim is backed by an actual before and after run against the same 200-session benchmark. We threw out several ideas that sounded reasonable on paper but measured worse in practice.
Built a genuinely usable interactive CLI on top of the same agent, so the system isn't just a JSON score. You can actually talk to it.

What we learned

The biggest lesson wasn't about retrieval algorithms. It was that the evaluation harness is part of the problem specification, and reading it as carefully as the spec document itself surfaces mechanics, like the wildcard "other" attribute or how customer replies parse constraint text, that no amount of retrieval tuning would ever reveal on its own. We also relearned a basic habit of good experimentation: test every change in isolation against the real benchmark, keep what actually measures better, and be honest, including in this writeup, about the ideas that didn't pan out.

What's next

Swap the heuristic reranker for a real LLM ranking call over the top candidates. The interface is already written as a drop-in replacement point.
Move from BM25-style keyword retrieval to real embeddings for the diversity track, to handle paraphrased or synonym-heavy browsing queries that keyword search can't catch.
Tighten the buying and intent override scenarios further, since they remain the two weakest categories compared to the near-perfect boundary score.