L-GPT

Inspiration

We profiled the data before writing any agent code, and two measurements shaped everything: the organizer's simulator builds every customer utterance from the target product's own catalogue record (every constraint is a verbatim substring of the target document), and targets come from real purchase records, so they skew heavily popular (median 6,846 reviews vs 12 for the catalogue). So this benchmark is string-provenance recovery, not semantic search — and we matched the estimator to the generative process instead of defaulting to dense retrieval.

What it does

L-GPT is a multi-turn shopping copilot over a frozen 50,000-product Amazon catalogue. Each turn it parses the shopper's message into catalogue-attested evidence, retrieves through a SQLite FTS5 conjunctive ladder, ranks by evidence coverage plus a popularity prior, asks the clarifying question with the highest measured information value, and discloses candidates. A change of mind resets rejection state; nothing shown-and-rejected is repeated.

Official result — organizer's evaluator, unmodified: TechnicalScore 0.9715 (HitRate@10 0.995, MTTC 2.2 turns) vs the published 0.1067 BM25 baseline. 0 LLM tokens, $0, fully offline, ~18 s for all 200 public sessions. Held at 0.87–0.95 across four self-built 800-session population shifts.

How we built it

A hybrid escalation ladder: cheap exact mechanisms run first, and learned components are reachable only when the cheap ones cannot see the answer — enforced by control flow, not confidence thresholds.

Recognition gate → exact template parsing (the entire scored path: Python stdlib + in-memory SQLite FTS5, no network, no key)
Unfamiliar wording → DistilBERT dialogue-act router + scaffolding tagger + exact catalogue span recovery
Unattested values → a hosted LLM deparaphraser (openai/gpt-oss-20b via Groq/OpenRouter/Cloudflare Workers AI) proposes a canonical term, and the catalogue must attest it before use — the model proposes, the catalogue disposes; no layer can invent a product
An optional LLM relevance filter demotes contradicting items for real-shopper wording

Every design decision is a numbered experiment with a recorded verdict — ~70 of them, negative results kept: dense retrieval (−0.047), cross-encoder reranking (−0.030), LLM listwise reranking (−0.027) and our own LLM relevance filter (−0.01/−0.02 on simulator traffic → shipped demo-only) all measurably lost to the exact core + popularity prior.

Challenges we ran into

Keeping the LLM honest was the hard part: given a candidate list it anchors to it, and an encoder "sanity check" on its proposals turned out inverted (AUROC 0.39) — so we inverted the design: the LLM only ever proposes text, and the frozen catalogue is the sole authority on what enters retrieval. The other challenge was not fooling ourselves: public-set tuning gains shrank ~71% on independent folds, so we froze the config and stopped optimizing.

Accomplishments that we're proud of

9× the baseline at $0 inference cost; a reproducible-from-clone repo (one command reproduces 0.9715, an integrity checker proves organizer files untouched); robustness measured on self-generated paraphrase suites (0.60 → 0.81 with the hybrid on); and an experiment log where rejected ideas are documented with numbers instead of deleted.

What we learned

Measure before believing. On this benchmark, popularity priors beat semantic models at ranking (57.4% vs 41.2% target-first), and a rate-limited A/B can produce a fake positive — our filter looked +0.002 until we re-ran it at full throughput and it was −0.01. Full report (architecture, cost/latency disclosure, limitations, per-member contributions) is REPORT.md in the repo.

What's next

Re-calibrating the popularity prior for real demand distributions (the inverse-popularity fold bounds that cost at −0.09), enabling the budget/brand questions real shoppers value (dead turns against the simulator, measured 0%), and growing the demo UI into a full shopping surface.