ByteSize: Knowing When to Stop Asking

Inspiration

Two kinds of shopping search fail, and they fail in opposite directions.

Filter walls make you translate what you want into someone else's taxonomy. You know you need a belt that will survive daily wear, and the interface asks you for buckle closure, width, material composition. Conversational assistants swing the other way: they interrogate you for turn after turn before showing you anything at all.

Both spend the same currency, and it is not compute. It is patience — the one thing a shopper will not give twice.

The Track 4 scoring rule agrees, and says so in arithmetic:

$$\text{TechnicalScore} = 0.50 \times \text{Hit@10} + 0.30 \times \text{MRR} + 0.20 \times \text{Efficiency}$$

$$\text{Efficiency} = \mathrm{clip}!\left(\frac{11 - \text{MTTC}}{10},\, 0,\, 1\right)$$

Every extra turn is taxed. So we stopped asking how should the agent rank? and started asking when does it know enough to rank at all?

What it does

A customer arrives with a vague message and an anonymized preference profile. The agent has at most 10 turns to surface the one product they actually want, out of a frozen catalog of 50,000 Amazon clothing, shoes and jewellery items.

Every turn resolves exactly one decision:

$$\text{ask} \iff \mathbb{E}\big[\text{gain from one more disclosure}\big] > \mathbb{E}\big[\text{gain from ranking now}\big]$$

When the evidence is insufficient it asks a natural clarification question. When the evidence is sufficient it commits to a ranked Top-10 and stops.

It converges in 2.75 turns against a budget of 10, finds the target in 100% of labelled public sessions, and does all of it at 0 tokens and $0 per session, with no network access required.

How we built it

Five deterministic stages, standard library only on the scored path:

Dialogue state — slots, scoped intent override (referenced replace, attribute replace, global reset), scenario detection.
Exact-evidence AND — category lock plus verbatim conjunction over disclosed slots. A filter that would empty the pool is skipped rather than applied.
Value-of-Information controller — insufficient evidence asks; sufficient evidence recommends. It keys on scenario, slots disclosed, whether the shopper has already signalled no additional preference, and pool size.
Popularity-first late fusion — popularity $1.0$, exact field-line match $0.35$, distinctive title phrase $0.15$, MiniLM cosine $0.1$. The dense signal is late fusion only: the conjunction builds the pool first, then cosine adjusts the order. It never participates in recall and never overrides popularity.
Optional listwise LLM rerank — implemented, measured, and shipped disabled, because it did not clear our holdout gate.

The controller deliberately never looks at how many turns remain. A controller that races the clock stops asking exactly when asking is most valuable.

What we learned

"Small pool" is not "ready to answer." This was the insight the whole submission turned on, and we had it backwards at first. A pool can collapse to five items because the shopper disclosed a category and a colour — and those five can be near-identical clones sharing one marketing template. Ranking them is a coin flip. One more question would have produced a distinguishing attribute and turned that coin flip into a rank-1 hit. Pool size measures how much is left, not how separable what is left actually is.

A protocol detail can be worth more than a ranking trick. The simulator only leaks the shopper's full intent text through one specific field. So the agent always asks on that field, while phrasing the visible question naturally around the attributes still missing. That is what makes the extra question cheap enough to be worth taking.

Optimizing rank-1 is not the same as optimizing Hit@10. The first result is the one people actually click. Our gain landed there, and it came with no Hit-rate cost.

Challenges we ran into

Resisting our own public-set score. Hit@10 of 1.000 on 200 inspectable sessions makes every new heuristic look brilliant. Several of ours improved the public number while quietly hurting held-out data. A score on 200 sessions we can read is not a prediction for 800 private sessions we cannot.

So we stopped optimizing and built two evaluation layers first: an ID-disjoint holdout of 200 sessions whose target ASINs never appear in the public set, and a random 800 split into 8 shards of 100, scored independently so we could see variance instead of one lucky mean.

That is where our strongest evidence comes from. Against the ablation baseline, the Value-of-Information controller delivers +60 rank-1 recommendations across 800 unseen, ID-disjoint sessions with zero Hit-rate loss, improving on 8 of 8 shards. A single-number improvement can be noise. The same direction on every shard is not.

Set	n	Hit@10	MRR	MTTC	TechnicalScore	Rank-1
Public (labelled)	200	1.000	0.954167	2.75	0.95125	184
Our ID-disjoint holdout	200	0.980	0.864845	2.885	0.911753	162
Random 800, 8×100 shards	800	0.97375	0.888018	2.8975	0.91533	672
Weak BM25 starter	200	0.125	0.068034	9.81	—	—

Killing our own work. We ran ablations on RRF fusion, BM25 over distinctive tokens, IDF and exclusive-term weighting, title-uniqueness scoring, catalog-provenance features, popularity-head guards, and listwise LLM reranking. None cleared the holdout gate, so none shipped. Every rejection is recorded with its numbers in report/methods.md. The discipline of not shipping a change that only helps the set you tuned on is itself part of the result.

Telling "runnable" apart from "score-equivalent." The agent degrades cleanly when the sentence encoder is unavailable, and it would have been easy to call that an optional enhancement. Measuring it honestly showed a real holdout drop from 0.980 to 0.975 — so we treat the encoder as a bundled correctness dependency and say so out loud.

Accomplishments that we're proud of

Zero marginal inference cost. 0 prompt tokens, 0 completion tokens, $0 per session. The scored path calls no LLM.
CPU only. No vector database, no training. The in-memory index over 50,000 items builds in ~8 s; 200 full sessions complete in ~35 s, roughly 0.2 s per session.
Offline by construction. With no environment variables set, no model backend is constructed and no network request is made. The encoder ships pinned at revision c9745ed1, so disabling network access does not change the score.
133 tests covering the agent contract, dialogue state, override scoping, response guards, retrieval, dense fusion, model fallback and submission packaging.
A demo video built by a script, not screen-recorded — so every number on screen is regenerated from the frozen transcript rather than typed into a slide.

What's next

The remaining oracle gap on our holdout is roughly $0.0026$. Analysis in report/disclosure.md and report/provenance.md shows it is concentrated in catalog-side clone disambiguation, not in the stopping policy. The next honest step is richer product-side evidence — not another stopping rule.

The mechanism itself is catalog-agnostic. The controller reasons about evidence sufficiency and slot disclosure, not about clothing. Swap the catalog and the attribute vocabulary and the same policy applies to electronics, groceries, or travel.

The narrow lesson is about 50,000 fashion items. The transferable one is that knowing when to stop asking is a modelable decision, and modelling it explicitly beats tuning a ranker.