• 

  An abstract for you!

• 

  One message in, exact product out: turn 1, rank #1, perfect 1.0000.

• 

  Customer changes their mind mid-chat — the agent re-ranks and still lands rank 1.

• 

  Official benchmark: 0.9801, MRR 1.0, all 200 targets at rank #1 in under 2 turns.

• 

  Seven-stage deterministic pipeline — no LLM, 100–200 ms per turn.

3W1Y shoppingbot

Inspiration

Ever frozen in front of a shopping site, drowning in 50,000 almost-right options? What you really want isn't a longer list — it's a soul mate who just gets you: one who asks exactly the right question and pulls the product you were picturing straight out of your head. That's 3W1Y shoppingbot.

And the science says this instinct is right: KDD 2016 framed recommendation as online learning where a couple of good questions beat a static model, and CIKM 2024 showed empirically that asking early outperforms recommending more. So we set ourselves one challenge for TechJam: out of a 50,000-item catalog, how few questions does it take to find the one?

What it does

3W1Y shoppingbot is a multi-turn conversational shopping agent. You describe what you want in free-form language; it asks the single most informative follow-up question, shows its current best pick, and treats "still talking" as feedback to try the next candidate. On the official 200-session public benchmark:

The official BM25 baseline scores 0.1067 — ~9.2x. Per-turn latency is 100–200 ms depending on hardware, after a one-time 30–60 s index build that is cached across runs. The pipeline spends zero LLM tokens — no LLM is called anywhere, by design.

How we built it

Our core insight came from a simple decision: instead of guessing how the customer behaves, we read the rules. In this challenge the "customer" is an open, deterministic program — given a target product, it always describes that product the same way and always answers questions by the same fixed rules. So we studied those rules and taught them to our agent. Now, for every candidate product still in play, the agent can predict exactly what the customer would answer if we asked about, say, color or material. That turns "which question should I ask next?" from a guess into a calculation: simulate every possible question against every remaining candidate, and ask the one that eliminates the most uncertainty — computed with the customer's real reply rules, not an approximation.

Around that, a seven-stage pipeline (in execution order):

1. Parsing — Tier-1 exact-template regex parser (0 tokens, <1ms), Tier-2 fuzzy parser for paraphrased or free-form language.
2. Dialogue state & routing — intent classification (buying/browsing/override/boundary) with a hand-set slot-update rule borrowed from GRU gating algebra: a "reset gate" clears conflicting slots on intent override, an "update gate" blends new evidence with old by parse confidence — no network trained.
3. Retrieval — category filter + cascading constraint matching with graceful backoff: a constraint only filters the pool when the survivors stay non-trivial; otherwise it degrades to a ranking feature instead of evicting the true target. A BM25 channel is fused in via Reciprocal Rank Fusion (the fusion utility is channel-extensible, e.g. for a future dense retriever).
4. Catalog-grounded belief scoring — for each retrieved candidate, replay how the customer would have described it and score how well the actual conversation matches; this ordered-observation likelihood is what pushed MTTC under two turns.
5. Belief-aware ranking — an interpretable linear scorer with the belief likelihood as its dominant signal, plus constraint coverage, intent-card overlap, BM25 and popularity; the quality/rating prior activates only once the customer has explicitly exhausted their stated preferences.
6. Guidance policy — expected-information-gain question selection with a confidence gate shaping the reply; most turns return both a clarifying question and the current recommendation.
7. Exposure policy — sequential Top-1 by default: a continuing session is implicit negative feedback, so each turn removes previously shown candidates and serves the next-best unseen one — any hit is a Rank-1 hit. A Top-10 fallback protects deep-ranked targets (normally on the final turn, or earlier once no informative question remains; a more aggressive widening schedule was tried and rejected — see Challenges).

Everything shown in the demo video — session replays, live benchmark runs, per-stage health — is served by the exact same agent that gets scored; Go watch and find out!

Challenges we ran into

• Products with identical attribute cards. Some catalog items are indistinguishable from disclosed constraints alone — an information-theoretic wall. Sequential exposure with implicit negative feedback turned an unsolvable ranking problem into a solvable exploration one; the quality prior only breaks remaining ties after preferences are confirmed exhausted, so it can't overpower incomplete evidence.
• "Deterministic" and "robust" turned out not to trade off the way we assumed. We expected the pure-deterministic build to be strictly best on exact official phrasing, with the fuzzy/BM25 layers only paying off under paraphrasing. Instead the hybrid build scored higher even on exact input (0.975 vs. 0.9706 at the time; the final belief-aware build reaches 0.9801), and the gap under the mildest paraphrase test wasn't marginal — the deterministic-only build collapsed to 0.0047. That result decided which config we actually submitted.
• Knowing when a signal is noise. BM25 score differences among identically-carded products are noise; popularity priors mislead before preferences are complete. We gated signal trust by dialogue state, and rejected an over-aggressive fix: widening exposure to a 3/5/10 schedule improved worst-case turn coverage but hurt paraphrase MRR by exposing some targets at rank 2 before they'd otherwise reach rank 1 — a documented negative result. The schedule we shipped instead (top-1 through turn 9, widen only at turn 10) is Pareto-safe: it extends worst-case deadline coverage from ranks 1–10 to ranks 1–19 without changing the rank or turn of any target the original policy already reached.

Accomplishments that we're proud of

• TechnicalScore 0.9801, MRR 1.0, and 200/200 Rank-1 sessions on the official benchmark — climbing from 0.9655 → 0.9751 → 0.9801 as we added sequential-exposure feedback and then the catalog-grounded belief update — ~9.2x the official baseline.
• A fully offline, fully explainable agent: 0 LLM tokens, 100–200 ms per turn, and a sub-200KB submission package with no third-party dependencies and no network calls.
• Robustness we can actually show a number for: 0.9800 under heavy paraphrase stress — essentially no drop — versus a documented catastrophic failure mode for the naive deterministic-only alternative (0.0047). The fallback layers aren't decorative.

What we learned

• Ask early, recommend late. One good question beats a longer list — and we can now put a number on it: the exact product is found in under two turns on average (1.995), worst case turn 6.
• Read the rules, don't guess them. The customer simulator is open, deterministic code — replaying its own reply logic per candidate is what turned question selection and belief updates from heuristics into exact computations.
• The same signal can be evidence or noise, depending on the moment. A high product rating is misleading while the customer is still stating preferences, but a fair tie-breaker once they say they have no more — so each signal is switched on and off by dialogue state.

What's next for 3W1Y shoppingbot

• Understand fully free-form language. The agent handles official phrasings and common casual variants today; truly unconstrained wording can still slip through. Next: a parsing layer that recovers key phrases directly from the sentence, plus an optional LLM-assisted tier where latency budgets allow.
• Already underway: the fuzzy parser now catches more everyday phrasing ("looking around," "on second thought," "whatever works"), while official templates keep their unchanged fast path.
• A real-store display mode, one config switch away. The competition setting shows one candidate per turn to maximize ranking metrics; real shoppers want to compare several items at once. Both modes already exist behind config flags (sequential_top1, late_turn_top_k), a hero pick plus a comparison list, no code changes needed.

Built With

• bm25
• information
• python
• regex
• sqlite
• theory