Conversational Shopping Agent

A multi-turn shopping agent that finds a hidden product in a 50,000-item catalogue in 2.8 turns on average, with no LLM, no API call and no token cost.

Repository: https://github.com/Antelyuu/techjam-conversational-search

What we built

The task gives the agent an anonymised customer profile and a short opening message. It then has up to ten turns to surface a hidden target product from a frozen catalogue of 50,000 Amazon clothing items, asking one clarifying question per turn as it goes.

On the 200-session public set:

metric	starter BM25	our agent
TechnicalScore	0.106710	0.945297
Hit Rate@10	0.125	1.000
MRR	0.068034	0.938657
MTTC (mean turns to convert)	9.81	2.815

Hit Rate is 1.000 on each of the four scenario types separately (Buying, Browsing, Intent Override, Boundary), so there is no headroom left in that metric. The agent uses no API key, makes no network call at inference time, reports 0 prompt and 0 completion tokens, and costs nothing per session.

Because there is no headroom left in that metric, the work we are proudest of does not show up in it at all. The public simulator quotes the target product's own text back at us verbatim, so we measured what happens when a customer paraphrases instead — and half our ranking went quiet. Closing that gap moved the paraphrased score from 0.696015 to 0.875897 and paraphrased Hit Rate from 0.805 to 0.965, while the table above moved by 0.0002.

We confirmed it against three structurally different paraphrase generators, two of them built specifically to be held out from tuning: one swaps vocabulary while keeping sentence structure, another reorders and reframes while keeping vocabulary. A gain that showed up only under the generator we tuned against would not have shipped, and twice a promising change was rejected for exactly that reason.

Two commands, no environment variables:

python3 -m evaluator.local_evaluator     # score all 200 sessions, ~35 s
python3 -m scripts.demo_session          # watch one conversation, turn by turn

pip install -r requirements.txt is optional. It enables the semantic ranking feature described above; without it the agent scores exactly the same 0.945297 and stays standard library only.

How it addresses the problem

The challenge asks for an agent that gets a vague shopper to the right product early and ranked high. Our approach came out of reading the evaluator's simulator rather than guessing at what it does.

The hidden intent card the simulated customer speaks from is not free text. The evaluator builds it by walking the target product's own features and details and taking whole values: one complete list element, or one complete key: value pair, normalised and clipped at 180 characters. So every constraint the customer can ever disclose is an exact member of a small set that the target product owns.

That changes what question you should be asking of a candidate. "Does this product's text look like what the customer said" is not discriminating, because near-duplicate listings all look alike. "Would this product have produced that exact string" is. Four things follow from it, and together they are most of the system. Two more exist because that whole insight is a property of this benchmark — a real shopper neither quotes the listing nor opens with the simulator's phrasing.

1. Slot ownership

We score each candidate on the share of the customer's disclosures it owns as whole values, weighted by how rare each disclosure is within the current candidate pool. "Machine wash cold" buried inside a competitor's longer bullet point tells you nothing. The same string standing alone as one of its feature values tells you a lot.

Measured over the catalogue: the target owns all 800 of its disclosable constraints, so requiring the match can never lose us a hit. 193 of those 800 are owned by exactly one product out of 50,000. Given the opening category plus two disclosures, the median consistent set is already down to a single product.

This is what got us past the near-duplicate listings that five phases of ranking work had failed to separate.

2. Search inside the stated category

The customer's opening line names the target's coarse category word for word, in every scenario, on turn 1. For Browsing sessions it is the only thing said before any question gets answered.

We reproduce the evaluator's category function exactly. Checked over the whole catalogue rather than a sample, it disagrees on 0 of 50,000 products and extracts correctly on 200 of 200 openers, which means the target is guaranteed to be inside the restricted set. The catalogue holds 1,115 coarse categories and the median target shares its own with 184 products, so a 400-deep pool now covers an entire category instead of 0.8% of the catalogue.

This is what fixed our last remaining miss. That session's intent card was entirely generic ("polyester", "Imported", "Zipper closure"), so every disclosure flooded the query with terms that tens of thousands of rows match. The target was never reaching the pool at all. It was a retrieval failure rather than a ranking one, which is why so much ranking work had never touched it.

3. Ask the open-ended question first

The simulator answers the other attribute with any undisclosed constraint, so its yield is always at least as high as any specific attribute's. We had it queued behind six narrower questions. 44 of the 200 sessions were not draining their intent card until turn 8 purely because of that ordering. Moving it to the front improves MTTC monotonically the whole way up.

4. Only return what the agent can defend

The evaluator ends a session as soon as the target appears anywhere in the returned list, and freezes the rank it appeared at. That makes a turn-1 list padded out to ten items a gamble: if the target happens to land at rank 7, the session ends there at a reciprocal rank of 0.14, and the later disclosures that would have lifted it to rank 1 never happen.

So while the agent is still narrowing the field it returns a single best candidate alongside its question, and the full ten once the constraints have identified something or the useful questions are used up. The widening conditions are measured rather than scheduled. If a customer paraphrases instead of quoting, no candidate owns anything, the policy detects that and turns itself off, returning ten as before.

5. Score meaning when the customer stops quoting

Everything above rests on the customer handing back the target's own strings. That is true of this simulator and it is not true of shopping. So we replayed all 200 sessions through a customer who rewords, and watched the agent fall from 0.945 to 0.696.

The feature that failed worst was the one that looked safest. Our token-overlap feature carried the second-largest weight in the table, and the argument for that weight was that a paraphrase would match nothing and it would fall silent. It does not: it scores partial overlap, so rewording fills it with whichever words happened to survive. Deleting it outright made the paraphrased score go up.

It is now replaced by the same question asked semantically — the cosine between what the customer said and each candidate's own field values, at exactly the granularity slot ownership uses. A shopper who says "something in cotton" instead of "Material: 100% Cotton" shares no token with the target and all of the meaning.

The part worth stealing is the gate. This feature is weighted an order of magnitude above slot ownership, which is right once exact matching has gone dark and wrong before it has — ungated, it cost us a hit on the verbatim set. So it is scaled by the share of disclosures that no candidate owns: it speaks exactly as loudly as the exact features are silent. That version is better on both sides at once, restoring Hit Rate 1.000 on the public set while beating the ungated form under paraphrase.

The model is voyage-4-nano — Apache-2.0 open weights run locally, so there is no API key, no network at inference, and no token cost. We chose it over MiniLM and bge-small on measurement, and it is the only one of the three whose catalogue artifact fits in a git repository: Matryoshka truncation to 256 dimensions plus int8 storage takes 268,564 vectors down to 66 MB for six millionths of quality.

6. Recognise the category however the customer asks for it

The same concern applies to the customer's first message. The simulator always opens with one of a handful of phrasings, so we originally recognised only those — and a real shopper who asks a question instead got no category at all, which widens retrieval from a median 184 products to all 50,000. The agent now finds the category by looking for it rather than for the phrasing around it, so Hey, do you have any Shoes Loafers & Slip-Ons? and hiya, after some accessories belts both work. It returns nothing rather than guessing when no category is named, because a wrong category loses the session while a missing one only widens the search.

Every feature's contribution to a candidate's score is recorded and printable via RerankedCandidate.explain(), so we can read off why something ranked where it did. That paid for itself early: our first reranker weighting scored worse than no reranking at all, and the per-feature breakdown is how we found which feature was responsible.

How we built it

Eight phases, each on its own branch, each closed out with a written decision record.

customer message
      ↓
ConversationOrchestrator   parse the opener, absorb the answer to our last
                           question, detect intent overrides, accumulate
                           disclosed constraints across turns
      ↓
retrieve()                 BM25 over SQLite FTS5, restricted to the stated
                           coarse category, 400-deep pool (plus an optional
                           dense route, off by default)
      ↓
rerank()                   deterministic 11-feature scorer, led by slot
                           ownership and phrase containment, with semantic
                           evidence taking over as they go quiet
      ↓
shortlist_size()           return only as many results as the evidence supports
      ↓
choose_attribute()         pick the question with the highest expected yield

Some notes on the engineering:

BM25 is SQLite's built-in FTS5 with per-field weights. No external search engine, no vector database, no index files to ship with the submission.

The reranker is an explicit feature checklist with fixed weights and no learned parameters. Every weight was swept and every sweep is written down, including the one we left in the table at 0.0 because knowing a feature was measured and found worthless is more useful to the next reader than deleting it silently.

Failure paths are all contained. A reranker exception falls back to the fused retrieval order instead of raising into respond(). A clarification failure asks nothing rather than costing the turn. An empty filtered search reruns unfiltered. A missing embedding artifact, a missing dependency or an artifact built from a different catalogue all disable the semantic feature rather than scoring against the wrong rows. Every degraded path prints its reason to stderr, so a bad run is visible rather than just quietly scoring lower.

237 unit tests, standard library only, in about 3 seconds. Several check our reconstruction of the card generator against the evaluator's own functions over a catalogue sample, so if the two ever drift apart the tests fail loudly instead of the feature silently going dead. The semantic feature's tests run against a synthetic artifact and an injected encoder, so the suite never downloads a model or reads the bundled 66 MB one.

Two of us tested each phase independently of whoever wrote it, and that is where the improvement agenda came from. A score tells you a phase is sitting at 0.63; it doesn't tell you what to do next. They worked out which cases were failing and which component was responsible. A target that never entered the candidate pool is a retrieval defect, not a ranking one, and the fixes for those two things have nothing in common. Making that distinction is what eventually located our last miss.

The dense route we built and then removed

Phase 3 built a full dense semantic route: MiniLM embeddings over all 50,000 products, benchmarked against bge-small, fused with BM25 by both RRF and weighted blending. It was worth +0.0355 at the time.

Phase 4 then changed what a query looks like. Once the customer starts answering questions by quoting constraint sentences out of the target's own text, and those quotes accumulate across turns, BM25 gets sharper on them while a single sentence embedding blurs them together.

configuration	before clarification	after clarification
lexical only	0.115573	0.687598
dense + RRF	0.145170	0.636669
dense + weighted	0.151089	0.636663

Switching the dense route off is worth +0.0509, and it wins or ties on every scenario. We turned it off but kept the code, the flag and the prebuilt artifact, because the result is about this particular query distribution rather than about dense retrieval in general. SHOPPING_AGENT_DENSE=1 brings it back.

docs/experiments/ holds fifteen decision records covering every measurement we took, including the things that didn't work: re-asking a productive attribute (−0.0078), keeping the customer's declines out of the BM25 query (−0.0246, because the query is thinner than it looks at a median of 6 distinct terms and the filler was acting as ballast), and a fused-score reranking feature that went nowhere.

Feasibility and cost

Measured over the full 200-session public set.

	default	`SHOPPING_AGENT_SEMANTIC=0`
model	voyage-4-nano, Apache-2.0, run locally	none
API key / network required	no	no
prompt + completion tokens	0	0
estimated cost per session	$0.00	$0.00
cold start (indexing 50,000 products)	4.3 s	4.1 s
per-turn latency	52 ms median, 121 ms p95	35 ms median, 75 ms p95
full 200-session evaluation	36 s	22 s
peak RSS	1.60 GB	0.78 GB
dependencies on the scored path	numpy, torch, sentence-transformers	none, standard library only
bundled artifact	66 MB (int8, 256 dimensions)	none

Both columns score 0.945297. The right-hand column is the agent without the paraphrase robustness; the left is what we ship. Two caveats stated rather than buried: the embedding model loads lazily, which makes one turn per run about 3.2 s, and it downloads its weights from the Hugging Face Hub on the first run only. If the organiser enforces a memory cap below ~2 GB, a per-turn timeout, or a fully sandboxed first run, SHOPPING_AGENT_SEMANTIC=0 gives up the paraphrase robustness and changes the benchmark score by nothing at all.

Being deterministic and offline means the agent holds up under the organiser's stated right to score submissions with network access disabled, and under CPU, memory and timeout limits. There is no rate limit to hit, no key to rotate, no vendor outage to survive, and no per-session bill that scales with traffic.

Adding the category filter also made a full evaluation pass 35% faster (41 s to 26 s), since BM25 now scans one category rather than the whole catalogue. The accuracy win and the cost win were the same change.

Every behavioural choice can be switched off in one step for auditing: SHOPPING_AGENT_SHORTLIST=0 (measures 0.885293), SHOPPING_AGENT_CATFILTER=0 (0.942229), SHOPPING_AGENT_SEMANTIC=0 (0.945297 here, 0.705561 paraphrased), SHOPPING_AGENT_DENSE=1, SHOPPING_AGENT_RERANK=0, SHOPPING_AGENT_CLARIFY=0. docs/SETUP.md lists the full set with what each one measures.

Why it matters outside the hackathon

Real catalogue search has the same shape as the problem we found here. Shoppers speak in fragments, catalogues are full of near-duplicate listings sharing the same vocabulary, and dense similarity ranks all of those listings about equally. A few things carry over:

Match structure, not prose. Retail catalogues are already structured, with attribute values, spec tables and care labels. Asking whether a product owns a stated value, weighted by how rare that value is among its live competitors, is a cheap discriminator that works precisely where embeddings tie. It needs no catalogue-wide statistics, since it calibrates against whichever candidates are actually competing.

A question is cheaper than a guess. Most of this system's score comes from clarification, and the highest-yield question is usually the open-ended one. Estimating expected yield per attribute before asking is a small change that has a large effect on how fast a session converges.

Precision over padding is also a product decision. An assistant that returns one item it can justify, and asks a question, is more trustworthy than one that returns ten and hopes. We are explicit below about where that trade is shaped by this specific metric.

Zero marginal cost changes what you can ship. A 52 ms agent with no API dependency — 35 ms with the embedding feature off — can sit in front of every search box for every user, including on hardware and in markets where per-query LLM inference isn't viable.

Development tools, APIs, libraries and data

Development tools

Claude Code, for implementation, code review and building the ablation harnesses
Git and GitHub, one branch per phase (phase/0-baseline-foundation through phase/8-semantic-evidence), reviewed before each merge
Python 3.13 (the agent itself requires 3.10+)
SQLite FTS5 for the BM25 index, built in memory at startup
unittest from the standard library, 237 tests
Our own measurement harnesses in scripts/: fusion, clarification and reranker ablations, weight sweeps, a target-survival audit, an offline rank replay that scores any shortlist policy in milliseconds and revalidates itself against the live evaluator on every run, and three independent paraphrase generators for measuring what happens when the customer stops quoting

APIs used

None. No LLM API, no external service, no API key, and no network call at inference time. We do ship one embedding model, but it is open weights running locally on prebuilt catalogue vectors, so the disclosure the rules ask for is the same either way: 0 prompt tokens, 0 completion tokens, $0.00 per session.

Libraries and frameworks

Standard library for the whole retrieval and dialogue path: sqlite3 (FTS5/BM25), re, json, dataclasses, math, unittest
sentence-transformers 6.0.0, which pulls in torch, transformers, numpy and huggingface_hub. Required only by the semantic ranking feature and the retired dense route. Without it the agent runs and scores an identical 0.945297, reporting once on stderr that the feature is unavailable — we treated "degrades rather than fails" as a hard requirement and it is covered by tests.

Models

voyageai/voyage-4-nano, Apache-2.0 open weights, run locally. This is the shipped model, used to score a paraphrased sentence against a product's field values. Chosen on measurement over the two below: 0.875897 paraphrased against 0.799906 for MiniLM and 0.825260 for bge-small, each compared at its own best weight because their cosine ranges differ enough that one shared weight would have understated two of the three. Its catalogue artifact is Matryoshka-truncated to 256 dimensions and stored int8: 268,564 vectors in 66 MB.
sentence-transformers/all-MiniLM-L6-v2, 384 dimensions, Apache-2.0. Chosen over BAAI/bge-small-en-v1.5 for the dense retrieval route (+0.013437), then measured again in Phase 8 and confirmed still ahead of it — but the route itself has been off by default since Phase 5, and re-measuring showed it still is not worth switching on.

Datasets and assets

Amazon Reviews 2023 from McAuley Lab, UCSD, Clothing_Shoes_and_Jewelry category. This is the organiser's frozen 50,000-product catalogue, joined on parent_asin, text and structured metadata only, no images or reviews or identifiers. See DATA_ATTRIBUTION.md.
200 labelled public development sessions from the organiser: 80 Buying, 80 Browsing, 30 Intent Override, 10 Boundary. Another 800 are held privately for final scoring.
A prebuilt voyage-4-nano value artifact, 268,564 × 256 int8, generated by us with scripts/build_value_embeddings.py and bundled so the semantic feature works offline in a fresh clone with no build step. It encodes one vector per distinct card value rather than per product, which is both the right granularity and 56% less encoding work.
A prebuilt MiniLM embedding artifact, 50,000 × 384 float32, generated by us with scripts/build_embeddings.py and bundled so the optional dense route works offline in a fresh runtime.
No manually labelled data. Every number quoted here came out of the organiser's deterministic evaluator or a script in scripts/.

Challenges

Getting stuck at rank 2. For most of the project the agent found the target in 190 of 200 sessions but only ranked it first 111 times, and the metric pays by position. We measured the cause instead of guessing: in 91-100% of the buried cases every text-based feature scored the target and the impostor identically. They were near-duplicate listings quoting the same words, so text similarity had nothing left to say. Slot ownership is what got past it, and it is still made of text. What changed is that it asks a structural question about the text.

Silent truncation, three separate times. A depth experiment above an internal pool cap once reproduced the capped depth while presenting itself as deeper. A lead-in regex capped at 60 characters was quietly dropping the hard constraint from 27 of 87 Buying openers, which is the most valuable thing a Buying session ever volunteers. Both were found in code review rather than by a falling score, because the score never fell. It just failed to rise. Every cap in the codebase now either sits far above its tuned value or carries the measurement that set it.

Constants expire. We swept pool depth five times and got a different answer each time, because each new discriminator changed what a deep candidate was worth. After the category filter it went completely flat, with depths from 200 to 1000 giving identical scores to six decimals. Our standing rule became: when the pool changes character, re-price everything that was tuned against the old one. The final re-sweep moved nothing by more than 0.0001, which we recorded anyway because otherwise someone would redo the work.

What we learned

Read the generator. Most of our score came from working out what the simulated customer actually is, a quoting machine walking a structured card, rather than from a better model. Two hours with local_evaluator.py were worth more than any embedding upgrade.

Measure before arguing. Our first reranker weighting, which followed the spec's own stated priority order, lost 0.047 against not reranking at all. Only the measurement told us the stated order was wrong.

Write down what failed. Several of our fifteen decision records exist mainly to stop a future reader from re-deriving a dead end. The rejected decline-suppression idea is documented specifically because it looks like a bug to anyone reading that code cold, and one whole record is five separate ideas that all measured to nothing.

Test the safety argument, don't just make it. For three phases we justified our largest feature weights with "if the customer paraphrases, these score 0 for everyone and the ordering falls back to what's underneath — the failure mode is silence, not noise." It was written in five places across three modules and it had never been measured. When we finally measured it, it was true of one feature, roughly true of a second, and false of the third: token overlap does not go to zero under rewording, it goes to a small number driven by whichever words happened to survive. At its weight that was not silence, it was noise with a loud voice, and deleting the feature outright raised the paraphrased score. The lesson we actually took is narrower than the one we started with: a comfortable argument about failure behaviour is a hypothesis, and it is usually cheap to test.

A weight tuned in one regime can be wrong in the other. The replacement feature's weight was swept against paraphrased replays, where the features it overrules are silent anyway — so nothing in that sweep could reveal that it swamped exact matching on the verbatim set. It cost us a hit before we gated it. Sweep in one condition, verify in the other.

Limitations

The shortlist policy is shaped by this metric. The evaluator breaks on the first hit and freezes the rank. Under a metric that scored the best rank across all turns, withholding results would be worth nothing. We think it is defensible as product behaviour, and nothing in the rules requires returning ten. But it is the one change a reviewer could fairly call metric-shaped, so we isolated it in a single module and documented it there. SHOPPING_AGENT_SHORTLIST=0 restores always-ten and measures 0.885293.

The agent still prefers a customer who quotes. Slot ownership, phrase containment and the exact category all depend on the simulator's verbatim behaviour, and mechanism 5 exists to cover exactly that. It closes most of the gap rather than all of it: 0.696015 to 0.875897 against 0.945297 when the customer does quote. Under paraphrase the target still sits at a median rank of 7 out of 400 where a quoting customer puts it at 2, so there is real ranking headroom left — a perfect reranker over the same retrieval would score 0.990300, and retrieval recall is already 1.000.

Opener parsing is regex-based and English-only. It reproduces the generator's message format exactly and is tested against it, but it parses one known format rather than doing general language understanding.

No LLM means no genuinely free-form conversation. The clarifying questions are templated. That costs nothing here, since the evaluator reads ask_attribute and never the prose, but a production version talking to real shoppers would need real generation.

Hit Rate is finished at 1.000 and can't rise. What's left is MRR at 0.938657, worth at most another +0.0184, and the reranker's entire adjustment half is now inert, so closing that gap would need a new discriminator rather than a retune. MTTC of 2.815 is bounded below by structure as much as by ranking, since an Intent Override session can't convert before its override turn arrives. The headroom that is actually open is not on this scoreboard, which is where we spent the last phase.

What's next

The paraphrased held-out split was the top of this list last time, and it is now built, shipped and the source of most of what we learned. What replaced it:

Give slot ownership the same treatment we gave token overlap. It is still whole-string equality and still one of the heaviest features. The semantic substitution that worked on its neighbour applies to it unchanged, and it is the largest single piece of paraphrase headroom we can name.

Build a harder held-out paraphraser. Our current held-out generators confirm that a gain transfers, but they are too easy to separate two good embedding models — all three we tested land within 0.0004 of each other on one of them. Any future model comparison needs a probe with more room in it.

Then: replace the templated questions with a small local generator so the conversation reads naturally without bringing back an API dependency; generalise slot ownership to a normalised attribute graph so it survives messier real catalogue data; and test the whole approach on a second product category, since every constant here was priced against clothing.