OPOYO

A shopper usually knows roughly what they want and not what it is called. OPOYO is built for that case. It works over a 50,000-item clothing catalog, and on each turn it shows ten products and asks one question. The session ends when the shopper's item turns up in those ten, or after ten turns.

We built it for TikTok TechJam 2026 Track 4. On the official evaluator it scores 0.842183, with Hit@10 of 0.915 and an average of 3.02 turns to a hit. The starter agent that ships with the kit scores 0.10671, so this is close to eight times the baseline, with no embedding models, no transformers, and no API keys. The whole agent is the Python standard library and SQLite. It installs nothing, and a full 200-session run costs $0 and about 25 seconds.

What it does

Every turn, OPOYO:

Puts the entire conversation so far into one BM25 query
Takes the top 400 products
Moves anything in the category the shopper named to the front
Cuts to ten
Sorts those ten by review count

Alongside that list it asks for other, an open slot rather than a named attribute like colour or size.

Retrieval runs two routes into one list:

A keyword route through BM25 across title, categories, features, details, store, and description
A category route taken from the shopper's opening line

The agent also reads intent from that opening line, tracks it in session state alongside every message and the named category, and asks a clarifying question on every turn rather than waiting for the candidate pool to get unwieldy. Under this scoring rule a question is nearly free. A hit on turn ten still scores 0.55 of a perfect one, while a miss scores nothing, so there is no turn worth saving by staying quiet.

That single path handles all four session types:

Buying. A hard constraint is stated immediately.
Browsing. Opens with a bare category and nothing else, which is why the first question carries so much weight.
Boundary. Refuses to answer once. The agent respects that instead of inventing a preference.
Intent override. The interesting case, because the obvious handling is wrong.

Thirty sessions have the shopper say "actually, ignore my earlier preference. What I need is X." The instinct is to clear what came before. But the hidden product never changes: behavior_for() draws the old value and the new one from the same intent card for the same item, so what the shopper just retracted is still true of the answer. We tested erasure and it drops those thirty sessions from 0.900 hit to 0.500. Keeping everything is the design, and that measurement is why.

Why asking openly beats asking precisely

One line in the simulator decides this. Its filter is:

attribute == "other" or classify_constraint(c) == attribute

A named label returns only constraints of that class, while other matches all of them. It releases at most two per turn, and every session holds exactly four, which puts everything on the table by turn three. No sequence of specific questions gets there faster.

Design decisions

BM25 over a neural reranker. Asking openly already extracts all four constraints by turn three, and the sessions we still miss are ones where the listing text is not unique to a single product. A cross-encoder cannot supply a detail the shopper never gave. A peer's MiniLM scored 0.64 on this task. BM25 also stays deterministic, so the same conversation returns the same ten products on any machine, and every score change below is attributable to the change that produced it rather than to sampling.
Popularity as a tiebreak, not a ranking signal. Sorting the scored ten by review count works. Sorting a wider window does not, and we have the numbers for both.
No withholding. The scoring locks rank at first appearance, so holding back a badly ranked early hit to land a better one later is arithmetically allowed. After popularity it stops paying. Three early bad hits remain, worth about 0.003, so it did not ship.

What we measured

The starter scores 0.10671, and two lines of it explain why:

It queries only the current turn, so it forgets what the shopper said.
It hardcodes ask_attribute to None, so the shopper answers "ask me about one specific attribute" and gives up nothing.

An agent that never asks never learns. Browsing sessions make that vivid, since the starter solves 0.025 of them and no boundary session at all.

Those two faults turned out to be independent, and fixing either alone left most of the gap.

arm	the single change	Hit@10	TechnicalScore
v1	accumulate the dialogue, still never ask	0.270	0.228414
v2	ask `other`, but stay stateless	0.560	0.496973
v3	both	0.875	0.750401

Accumulating alone solves 17 sessions.
Asking alone solves 75.
48 need both.

Asking is the larger lever, which we did not expect. Memory has nothing to store until a question produces an answer.

After v3, 175 of 200 sessions hit, and 42 of those hits arrived at rank 5 through 10. That number changed what we worked on. BM25 had already found the product, so the position it arrived in was the problem rather than the query. The evaluator locks the recorded rank at the first turn the target enters the ten, so a better list afterwards recovers nothing.

Sorting just those ten by rating_number holds hit rate at 0.875 and lifts MRR from 0.540 to 0.716, taking the score to 0.803243. On the 42 buried hits alone, MRR moves from 0.145 to 0.846.
Ordering the top 400 by popularity before cutting to ten does the opposite, dropping sessions we had already solved from 102 to 74. Popularity works as a tiebreak inside a set retrieval has already narrowed, and as a retrieval signal it is actively harmful.

The last gain was sitting in the opening line. Every session begins with I'm looking for {category}, and the final-evaluation FAQ froze those templates. Moving that category to the front of the top 400 before cutting to ten raises Hit@10 from 0.875 to 0.915: eight sessions gained, none lost. That is the shipped agent.

What the score can and cannot measure

A shopper who cannot name a product can still react to one, so showing ten concrete items is a way of narrowing the search rather than a guess at a fixed answer. Saracay, Schmidt and Guestrin make that argument in Beyond expert users (Stanford, June 2026) and built CoShop to test it. Five frontier models stayed under 56% accuracy over five turns there, failing by never expanding what the shopper knew.

This harness measures something narrower. The function that writes the shopper's next line is customer_reply(sample, ask_attribute, disclosed, boundary_used), and the ranking is not one of its arguments, so nothing the agent shows can change what the shopper wants.

Kim et al. name the pattern in Stop Playing the Guessing Game! (EMNLP 2025 Findings).
τ-Rec (RecSys 2026) reveals constraints through the same kind of channel.
ConvApparel (EACL 2026) measured how far these simulators sit from real shoppers, in this same apparel domain.

So OPOYO returns two things per turn:

The ranking and the question play the scored game.
The message field presents those ten products as examples for a person to react to.

We built both, and we would rather say which half the number measures than blur them together.

Results

Unmodified official evaluator, 200 public sessions, run twice with identical output.

Hit@10	MRR	MTTC	TechnicalScore	tokens	cost
0.915	0.750276	3.02	0.842183	0	$0

About 40 ms a turn on a laptop CPU
No GPU
No network call at scoring time
The variants we rejected are checked into the repository next to the run that beat them

Limitations

Reordering by review count is worth 0.053 of the final score, and it may partly be measuring the evaluation set rather than the shopper. Amazon targets sampled leave-last-out skew popular, which makes a popularity prior resemble skill. Cañamares and Castells set out when it is genuine signal and when it is an artifact of the test collection in Should I Follow the Crowd? (SIGIR 2018).
Seventeen sessions still miss. A belt described as leather, 100% leather, imported, buckle closure matches thousands of belts, and no ordering recovers a detail the shopper never gave.
The construction policy is not testable on this harness, which needs a target-free protocol of the kind Kim et al. propose.
These numbers are the public 200. The private set is released after the deadline.

Given more time, the first thing we would test is personalization. Each session ships an anonymized user_profile, and we have not yet measured whether its preference tags sharpen the query or add noise.

Built with

Development tools. Python 3.10 on a laptop CPU, edited locally. Official public set: python3 -m evaluator.local_evaluator. Agent tests: python3 -m unittest discover -s tests -v. No Colab, no Jupyter, no GPU machine.
APIs. None. No OpenAI, no Gemini, no Hugging Face Inference, no paid ranking endpoint. Scoring makes no network call.
Libraries and frameworks. Python standard library only: json, re, sqlite3, pathlib. Retrieval is SQLite FTS5 BM25 with tokenize='unicode61 remove_diacritics 2'. Tests use unittest. Nothing to install.
Datasets and assets. Frozen TechJam participant kit, derived from Amazon Reviews 2023 (McAuley Lab, UCSD), category Clothing_Shoes_and_Jewelry:
- 50,000-product catalog
- 200 labeled development sessions: 80 buying / 80 browsing / 30 intent override / 10 boundary
- Official local evaluator, unmodified
- Attribution in DATA_ATTRIBUTION.md