Hazelnut

Hazelnut is a conversational shopping agent, given a short customer message, the agent returns ranked catalog recommendations, surfacing the target product as early and as highly ranked as possible.

Comment

Inspiration

Conversational shopping is fundamentally different from one-shot search. A customer's intent arrives incrementally, preferences can change mid-session, and the agent must decide both what to retrieve and when it knows enough to recommend confidently.

While analysing the challenge's data-generation process, we found an important property: disclosed requirements are grounded directly in product metadata. That changed our retrieval strategy. Once explicit constraints arrive, precise structured evidence becomes more valuable than broad semantic similarity.

Our first implementation leaned heavily on dense semantic retrieval. We measured rather than assumed its value: fusing dense vectors into the ranking was monotonically harmful as the fusion weight increased. Semantic neighbours were often plausible, but wrong; exact constraint evidence was more discriminative. Dense retrieval therefore remains useful for cold start, before useful constraints are disclosed, while lexical and structured evidence dominate once the customer's intent becomes specific.

That principle shaped the rest of Hazelnut: use the simplest signal that the evidence supports, measure every architectural change, and keep only what survives evaluation.

What it does

Hazelnut is a conversational shopping agent for multi-turn product discovery. Given a short customer message and up to 10 turns, it:

  1. identifies whether the customer is buying or browsing;
  2. extracts the active category and constraints;
  3. retrieves candidates from a 50,000-product catalog;
  4. reranks them using explicit constraint coverage and structured product evidence;
  5. detects when the ranking is ambiguous and asks for clarification;
  6. handles changes of mind through active/superseded intent state; and
  7. adapts how many recommendations it surfaces to the interaction surface.

Architecture

Customer
   |
   v
Intent + Session State
   |
   v
Hybrid Retrieval
(category + lexical + dense cold start)
   |
   v
Structured Reranking
(constraint coverage -> signature agreement -> retrieval score)
   |
   v
Uncertainty Detection
   |----------------|
   v                v
Clarify          Recommend
   |                |
   |-------+--------|
           v
      Session Loop

Hazelnut continuously turns conversation state into intent-aware retrieval, structured evidence reranking, uncertainty-aware clarification, and deployment-aware recommendation exposure.

Public-Set Results

Evaluated on the organizer's SHA256-verified frozen 50,000-product catalog and all 200 public sessions.

System Hit@10 MRR MTTC ↓ TechnicalScore
Organizer BM25 0.125 0.068 9.81 0.107
Hazelnut 1.000 0.996 2.40 0.9707

9.1× the organizer baseline · 100% Hit@10 · 0 LLM calls · no network · no credentials

Beyond the public benchmark, we also use held-out sessions and same-session paired evaluation to test whether improvements generalize beyond the 200 public sessions.

How we built it

Pillar I --- Intent-aware hybrid retrieval

router.py classifies buying vs browsing, extracts the category, and accumulates disclosed constraints across turns.

retriever.py then filters to an exact category bucket recomputed from catalog data and scores candidates using IDF-weighted constraint coverage, verbatim-phrase bonuses, reverse-containment matching, and a small popularity prior for weak ties.

dense.py provides an in-memory TF-IDF -> randomized-SVD LSA index implemented in NumPy. Dense retrieval is deliberately concentrated at cold start, where semantic similarity is useful before explicit requirements arrive.

Once the customer reveals near-verbatim product evidence, lexical matching becomes more precise than semantic neighbourhood similarity. Our dense-fusion ablations confirmed that distinction.

Pillar II --- Structured evidence reranking

A major improvement came from diagnosing where our remaining failures occurred.

Several misses initially looked like candidate-recall failures. Per-session tracing showed that many targets were already retrieved but buried among products sharing generic constraints such as cotton, 100% Cotton, Imported, and Button closure.

Simply widening the reranking pool was not a reliable solution: it occasionally rescued a deep target, but more often disturbed sessions that were already correct.

We therefore changed the ranking signal, not merely the ranking depth.

The local reranker prioritizes:

1. distinct explicit-constraint coverage
2. structured signature agreement
3. existing retrieval score

For candidates satisfying the same number of requirements, Hazelnut compares a compact signature derived from ordered catalog evidence. A candidate receives a stronger tie-break signal when its structured evidence better explains the sequence of requirements disclosed during the conversation.

Crucially, this signal is subordinate to explicit user intent: a candidate satisfying fewer requirements cannot outrank one satisfying more simply because its signature looks better.

Pillar III --- Dialog state and intent override

Hazelnut maintains shared session state containing current intent, category, disclosed constraints, per-attribute slots, superseded constraints, and exposure/rejection history.

Intent override required special treatment. If a customer changes a preference, hard-deleting the old value throws away potentially useful retrieval evidence, while treating it as fully active produces incorrect conversational state.

Our solution is active vs superseded intent: a previous preference is retained as historical retrieval evidence while becoming inactive for current dialog state. The harder erase-on-override behavior remains reproducible behind a flag but is not the default because it measured worse.

Pillar IV --- Ambiguity-aware clarification

An early implementation used fixed-turn withholding. It scored well, but it could not distinguish genuine uncertainty from a confident ranking.

We replaced that schedule with a ranking-margin ambiguity signal. When the leading candidates are too close, the agent treats the state as uncertain and can clarify before committing. When the ranking is confident, it proceeds normally.

This same uncertainty signal informs proactive clarification and conservative early recommendation exposure.

Pillar V --- Runtime feedback and self-evolution

We implemented and evaluated context distillation, cross-session memory, item-level rejection handling, and aspect-level negative feedback grounded in Bi et al. (CIKM 2019).

These mechanisms were ablated independently rather than enabled simply because they sounded sophisticated. Aspect-level negative feedback survived final re-verification and ships enabled. Other mechanisms that did not produce reliable gains remain behind flags as reproducible experiments.

Beyond the brief --- evaluation infrastructure

We built two held-out session generators, paraphrase-robustness testing, per-session failure tracing, popularity/constraint-count slicing, provenance logging, and paired_compare.py.

The paired comparison tool evaluates two configurations on the exact same sessions and reports wins, losses, ties, largest movers, and an exact sign test. This repeatedly prevented us from promoting changes that looked good on one aggregate score but failed to generalize.

Challenges we ran into

The hardest part of Hazelnut was not implementing more mechanisms. It was learning which mechanisms deserved to survive.

Dense semantics were not the answer everywhere

Our initial instinct was to lean heavily on semantic retrieval. But once explicit constraints appeared, exact metadata evidence consistently outperformed semantic similarity. Dense ranking fusion became worse as its weight increased.

Lesson: semantic similarity is useful for exploration; it is not automatically the right signal once the customer becomes specific.

LLM reranking did not beat the local reranker

We built and evaluated four Claude reranking variants: blanket listwise, ambiguity-gated listwise, pairwise top-2, and pairwise top-3. Every variant measured equal to or worse than our small deterministic local reranker.

Lesson: on highly structured evidence, model size is less important than calibration, intervention width, and preserving already-correct rankings.

Wider reranking attacked the symptom, not the cause

Tracing complete misses showed that many targets were already inside the candidate pool but ranked too deeply. We tested a wider reranking pool, which occasionally rescued a miss but introduced smaller regressions across more already-successful sessions.

That led to the more useful question: why are these targets deep in the first place? The answer was generic constraints creating flat rankings, which motivated structured evidence reranking.

Our hardest lesson: re-verify the measurement itself

Aspect-level negative feedback initially replicated at +0.017 across three held-out draws. After the exposure policy changed, it appeared to reverse to -0.006, so we disabled it.

A later audit found that the second measurement had accidentally evaluated the feature with a different exposure configuration than its baseline. Once corrected, the effect became positive again and statistically significant on two held-out draws.

That changed our engineering rule from re-test every mechanism after the baseline changes to the stronger rule:

Re-verify the measurement, not just the claim.

Every major mechanism therefore remains reproducible behind a flag, including experiments that failed.

What we learned

1. Diagnose before optimizing. A miss is not automatically a retrieval-recall problem. Our traces showed that many failures were already retrieved and instead needed a better discriminative ranking signal.

2. More AI is not automatically better. Four LLM rerankers failed to outperform a compact local reranker. The final scored path therefore uses zero LLM calls.

3. Current intent should dominate weak historical context. We tested profile personalization and other memory mechanisms. Improvements on selected cases did not survive held-out evaluation strongly enough to justify enabling them by default.

4. Architecture changes can invalidate old ablations. The meaning of a rejection changed when our exposure policy changed. Previously validated mechanisms therefore had to be measured again against the new baseline.

5. Reproducibility is a feature. We record the commit, dataset, catalog, ranking flags, exposure mode, and other ablation settings with each run. Paired evaluation on identical sessions became our default way to decide whether a change was real.

Deployment-Aware Recommendation Policy

Separating ranking quality from presentation strategy

Hazelnut deliberately separates what to recommend from how recommendations are presented.

The same ranked candidates may need to be surfaced differently depending on the interface. A voice assistant or sequential chat naturally presents products one at a time, while a web grid or app carousel allows several candidates to be compared in parallel.

Because recommendation exposure is decoupled from retrieval and reranking, Hazelnut can adapt to these surfaces without retraining or replacing the underlying ranking model.

Deployment modes

Mode Best suited for TechnicalScore
Walk (default) Sequential chat / voice 0.9707
--no-walk Web grid / app carousel 0.9580
--no-exposure-gate Ranking-only ablation 0.9151

All three modes use the same retrieval and reranking pipeline; only the recommendation-exposure policy changes. Hit@10 remains 1.000 across all three evaluated modes.

  • Walk (default): surfaces one strong recommendation at a time, matching sequential interfaces such as chat or voice.
  • --no-walk: allows multiple candidates to be compared in parallel, better representing a web grid or app carousel.
  • --no-exposure-gate: disables recommendation-exposure controls and evaluates the underlying ranking through full-page outputs.

The --no-exposure-gate result is particularly important for interpreting our headline score. It disables recommendation-exposure controls and evaluates the underlying ranking through full-page outputs. The difference between this result and the default therefore reflects the contribution of conversational turn management, rather than an improvement in retrieval itself. We report both deliberately to separate ranking quality from presentation-policy gains.

For a production system, we would make this adaptive: infer the surface from client capabilities and select the appropriate presentation policy per session, so a customer using a smart speaker and one using a desktop grid each receive recommendations in the format their interface supports.

What's next

The most important next step is testing Hazelnut against genuinely free-form customer language rather than deterministic simulator templates.

We would also add slot decay so stale preferences lose influence over long real-world conversations, richer surface-aware presentation policies, robustness evaluation on real paraphrases/noisy product language, and selective LLM assistance only where free-form language creates ambiguity that structured evidence cannot resolve.

Built with

Core stack

  • Python 3.10+
  • NumPy --- sole required third-party dependency
  • in-memory TF-IDF, randomized SVD, and cosine similarity
  • Python unittest
  • git / GitHub
  • VS Code

The submitted scoring path requires:

GPU:          no
Network:      no
API key:      no
LLM calls:    0
API cost:     $0

No PyTorch, Hugging Face, scikit-learn, FAISS, or vector database is required.

Optional LLM experiments

The repository includes four Claude Opus 5 reranking variants used during development. All were evaluated against the local reranker and measured equal or worse, so none is enabled in the submitted path.

They remain in the repository as reproducible negative experiments, not as dependencies of the final system.

Data

We use the organizer's frozen 50,000-product catalog and 200 labeled public sessions derived from the Amazon Reviews 2023 Clothing_Shoes_and_Jewelry data.

  • no external training data;
  • no manual labeling;
  • no catalog mutation.

Reproducibility

# Default conversational policy
python3 tools/run_eval.py --agent pipeline

# Batched / grid-style presentation
python3 tools/run_eval.py --agent pipeline --no-walk

# Ranking-oriented exposure ablation
python3 tools/run_eval.py --agent pipeline --no-exposure-gate

For A/B experiments on identical sessions:

python3 tools/paired_compare.py results_A.json results_B.json

Every evaluation result records provenance including the commit, dataset, catalog path, reranker, dialog mode, exposure policy, and experimental flags.

Links

Repository: https://github.com/gaozilin2005/hazelnut-bubble

Demo video: https://youtu.be/JrU48Vda4E4

Built With

+ 5 more
Share this project:

Updates