Beyond Retrieval

Iteration Evaluation

Inspiration

Shopping conversations are messy. People browse vaguely, drop hard constraints mid-sentence, change their mind on turn 4, or say "I don't really care about that." Most search systems treat every message like a keyword query and hope similarity is enough.

When we read the TechJam challenge spec, one detail stood out: the customer is not a free-form LLM — it is a deterministic simulator that only reveals information when you ask the right structured question (ask_attribute). The official BM25 baseline never asks anything and scores 0.107.

That made us ask a different question:

Instead of "which product looks most like what the user said," what if we ask "which product, run through the same generative process as the simulator, would produce exactly the words we heard?"

That reframing — from retrieval to inverse inference — became the foundation of Beyond Retrieval.

What it does

Beyond Retrieval is a multi-turn shopping agent that finds a hidden target product in a frozen 50,000-item e-commerce catalog within 10 dialogue turns.

On each turn it can:

reply in natural language,
ask a structured clarification question (category, material, color, style, other, etc.),
and return ranked product recommendations — or do all three at once.

It handles all four official session types:

Buying (40%) — hard constraints appear early
Browsing (40%) — vague opening, needs active exploration
Intent Override (15%) — preferences flip on turn 3–4 without changing the target product
Boundary (5%) — the customer may have no preference for a requested attribute

Results on the public dev set (200 sessions):

Metric	Score
TechnicalScore	0.9802 (ceiling ≈ 0.983)
Hit@10	1.000
MRR	1.000
MTTC	1.99 turns
LLM API calls	0
Mean latency	78.8 ms / turn

The agent runs fully offline — Python standard library only, no API keys, no network required.

How we built it

We iterated in three phases over Baseline → v1.0 → v1.5 → v2.0:

Phase 1 — Break the silence (v0.2 → v1.0)

The baseline's 12.5% hit rate was not a ranking problem — it was an information problem. The simulator replies "Ask me about one specific attribute" until you actually ask.

We built a dialogue state machine: slot extraction, typed questioning, ask other first to capture long distinctive constraints, Intent Override memory (don't wipe slots on override), and exclusion of previously shown non-hits. This alone took us from 0.107 → 0.882 with 100% Hit@10.

Phase 2 — Push retrieval to its ceiling (v1.1 → v1.5)

With full hit rate achieved, the bottleneck became rank and turn efficiency. We added fine-grained reranking and strategic delayed submission — sometimes waiting one turn to gather constraints beats locking in rank 8 on turn 1. Score reached 0.915, but further tuning plateaued.

Phase 3 — Change the paradigm (v2.0)

We replaced similarity ranking with Bayesian inverse inference:

$$\log P(\text{target} = p \mid \text{dialogue}) = \log \text{Prior}(p) - \sum_t \text{penalty}_t(p)$$

Four components in starter/:

user_model.py — byte-faithful replica of the customer generative process
catalog_index.py — intent cards, inverted indexes, and a rating-count prior ($P(\text{target}) \propto \text{rating_number}$)
belief.py — posterior over 50k candidates using positive, negative, and elimination evidence
policy.py — optimal question selection + sequential single-guess submission via dynamic programming

Instead of submitting a batch Top-10 every turn, v2.0 submits one candidate at a time — because rank 1 MRR is worth far more than saving one dialogue turn.

Stack: Python 3.10+, official evaluator harness, unittest fidelity tests, optional React/Vite dashboard for version metrics visualization.

Challenges we ran into

The simulator is a black box at runtime. We never see intent cards, ground truth, or internal simulator state — everything must be inferred from public rules and dialogue structure alone.

Asking is necessary but not sufficient. v1.0 achieved 100% hits, but MRR stayed at 0.695 because early vague submissions locked mediocre ranks. Balancing when to ask vs when to submit required modeling the scoring formula directly, not gut feel.

Retrieval has a hard ceiling. Entropy-based questioning (v7) and profile-guided branching (v8) both hurt performance. Ablations showed degrading v2.0 back to batch Top-10 drops score to ~0.91 — essentially v1.5's limit. We had to change the paradigm, not tune weights harder.

Intent Override is continuous, not a reset. When the customer says "ignore my earlier preference," the target product does not change. Wiping session memory on override was one of our earliest costly mistakes.

Offline evaluation constraint. Organizer policy may disable network access in final scoring — so every design choice had to work with zero external API calls.

Accomplishments that we're proud of

0.107 → 0.9802 — a 9× improvement over the official baseline, within ~0.003 of the theoretical score ceiling
Perfect ranking — MRR 1.000 on all 200 public sessions; every hit landed at rank 1
Sub-2-turn efficiency — average first hit at 1.99 turns across Buying, Browsing, Intent Override, and Boundary
Zero cost at runtime — no LLM, no tokens, no pip dependencies; 191 MB memory, 78.8 ms mean per-turn latency
Mechanism over brute force — understanding the customer's generative process beat expensive general-purpose models
Fully reproducible — public evaluator, ablation scripts, frozen snapshots (snapshots/v1.5, snapshots/v2.0), and architecture docs included in the repo

What we learned

Questions are the highest-leverage action in conversational search — if the environment only discloses information when asked, retrieval quality is irrelevant until you ask correctly.
Misses are free evidence — if a session continues after a Top-10 submission, those 10 products are definitively not the target. Treat failed rounds as hard eliminations, not noise.
The scoring function should drive the policy — one extra turn costs 0.02 efficiency points; rank 8 → rank 1 gains +0.26 MRR points. Decision theory beats heuristic timing.
Know when to stop tuning and change frameworks — v1.5 exhausted the retrieval paradigm; the +0.065 gain in v2.0 came entirely from reframing the problem.
Faithful simulation beats fuzzy similarity — a deterministic customer model you can invert is more powerful than embedding distance when the ground-truth process is known and structured.

What's next for Beyond Retrieval

Real-world conversations — adapt the inverse-inference framework beyond the competition simulator to open-domain human chat
Stronger robustness — our paraphrase/foreign-intent stress tests drop to ~0.83–0.89; better normalization and fuzzy constraint matching would help
Explainability — show users why each candidate was eliminated, not just the final recommendation
Hybrid tie-breaking — keep the zero-token core, add a lightweight neural reranker only among posterior ties on ambiguous sessions
Scale beyond 50k — explore hierarchical priors and approximate belief updates for larger catalogs