Verbatim — Grounded Conversational Shopping Agent

The task: find one hidden target among 50,000 products, within 10 conversation turns, from a frozen catalog.
Our core observation: every customer constraint is text derived from the target's own catalog entry — so we invert that derivation.
Official 200-session evaluator, live: HR@10 = 1.000, MRR 0.9729, 2.47 turns — TechnicalScore 0.962475 vs the 0.107 baseline.
A real captured session: vague opener, two open questions, then the true target returned at rank 1 on turn 3.
The confidence gate: an expected-value rule (rank gain worth 0.27 vs 0.02 per question) decides when to ask and when to answer.
Paraphrase stress test — every utterance reworded: 0.94, up from 0.571 before hardening. Robustness for the private 800 sessions.

Inspiration

The kit's weak BM25 starter scores 0.107 — it treats a conversation as a bag of words. But the challenge's data design hides a stronger signal: every customer constraint is text derived from the hidden target's own catalog entry. We built an agent around inverting that derivation, and around a simple economic fact of the metric: a better rank is worth up to 0.27 per session, while one more clarifying question costs only 0.02. That asymmetry — ask when evidence is thin, answer the moment it is decisive — is the whole design.

What it does

A multi-turn shopping agent that routes buying vs. browsing intent, keeps a structured constraint state (including explicit intent-override and boundary handling), asks confidence-gated clarification questions while the candidate pool is still ambiguous, and answers with a full ranked top-10 the moment the evidence pins the target. On the official 200-session public evaluator it scores TechnicalScore 0.962475 (HR@10 1.000, MRR 0.972917, MTTC 2.47) against a 0.10671 baseline and a 0.9922 theoretical public-set ceiling. Under our paraphrase-stress harness — every simulator utterance reworded, constraint values verbatim — it holds 0.93–0.95 across runs (up from 0.571 before hardening), our robustness argument for the private 800 sessions.

Why this matters beyond the leaderboard. Conversational commerce lives or dies on one product decision: when to ask a question versus when to show results. Ask too much and customers leave; show too early and they scroll past junk. Our confidence gate answers that with an explicit expected-value rule instead of a vibe, and it converts customers in ~2.5 turns with zero irrelevant recommendation lists. Because every recommendation is backed by verbatim evidence from the customer's own words, the agent can always show why an item was recommended — grounded, explainable, and with no hallucinated products by construction. And because the core is deterministic, offline, and zero-token (milliseconds per turn on a laptop CPU), the marginal cost per session is effectively zero — the cost structure that makes an assistant-per-session viable at marketplace scale, with an LLM reserved only for the one slot where it measurably pays.

How we built it

Three deterministic evidence tiers, all offline: Tier A inverts the kit's constraint-derivation logic into a card-to-products index for exact O(1) evidence lookups, accelerated by Aho-Corasick automata over every derivable catalog string so evidence is found verbatim inside any phrasing; Tier B scores paraphrased constraints by substring presence over the category pool; Tier C is field-weighted BM25 over a SQLite FTS5 index of the full 50k catalog. A confidence gate (minimum exact evidence, unique-dominance detection, customer-exhausted shortcut, turn-7 safety valve) decides each turn whether to keep clarifying or to fire. A two-layer parser (exact templates plus a universal verbatim-value extractor) survives rewording.

Stack: Python 3 with the standard library (SQLite FTS5 included) plus one lightweight C extension, pyahocorasick, with a pure-stdlib fallback path. No external APIs, no network access, no LLM in the scored path — zero token cost, reproducible under offline final scoring. Data: only the frozen competition kit (50,000-product catalog and 200 public sessions, derived from Amazon Reviews 2023, McAuley Lab UCSD — see DATA_ATTRIBUTION.md). We also built a paraphrase-stress fork of the official evaluator and a unittest suite covering the API contract and dialog behavior.

Challenges we ran into

Scoring well on fixed templates is easy; making the score survive rewording was the real work. Our first stress run collapsed to 0.571 and exposed the agent as a template-matcher. The redesign was a whack-a-mole campaign we had to fight with measurements, not intuition: normalization-tolerant matching fixed paraphrases but introduced false-positive evidence; broad single-word cards ("spandex", "down") opened the recommendation gate a turn too early; case-variant automaton keys silently overwrote each other and dropped real evidence. Each fix was validated on three boards at once (official / reworded / value-rewritten) across an 11-configuration sweep. The hardest call was disciplinary: an LLM-driven recall extension measured net-negative under stress, so we kept the deterministic core and confined the LLM to an optional, env-gated rerank slot.

Accomplishments that we're proud of

0.962475 official — 97% of the nominal 0.9922 ceiling and ≈99% of the feasible optimum: we proved the ceiling unreachable (intent-override sessions cannot legally convert before turn 3), so we know we are ~1 point from optimal play.
HR@10 = 1.000: all 200 public sessions find the target, in 2.47 turns on average.
A robustness recovery of +0.37 (0.571 → 0.93–0.95) under full rewording, with zero regression on the official board — evidence the agent understands customers, not just the simulator's templates.
A reusable evaluation practice: the paraphrase-stress fork of the official evaluator is a pre-deployment robustness check any dialog-agent team could adopt as-is.
Honest measurement: stress scores are reported as a per-run band, the deterministic official score to six decimals, and every design claim in our report is backed by an ablation in the repo's history.

What we learned

Measure your robustness before the judges do — a score that only holds on fixed templates is an overfit, and one stress run told us more than any refactor. Turn the metric's economics into policy: deriving the gate from expected score value (0.27 vs 0.02) beat every hand-tuned threshold we tried. And on grounded tasks, verbatim evidence beats semantic guessing: the disciplined move was proving where an LLM helps before spending tokens, not assuming it helps everywhere.

What's next for Verbatim — Grounded Conversational Shopping Agent

An LLM Tier D semantic reranker for handcrafted, non-derivable constraint values (the slot is already built and env-gated); attribute-targeted question selection for human customers, replacing the simulator-optimal open question; and an A/B-testable ask-vs-show policy — the confidence gate's expected-value rule, recalibrated on real conversion data instead of the competition metric.