ShopSIFT

🛍️ShopSIFT turns evolving shopping intent into ranked product recommendations through constraint tracking, adaptive questions, multi-route retrieval, and guarded local reranking.

Comment

The proposal

ShopSIFT treats conversational shopping as active exact-item retrieval under evolving constraints. It maintains a Counterfactual Constraint Ledger, asks score-aligned information-value questions, combines facet-aware multi-route retrieval with catalog-derived intent projection, and applies guarded, traceable reranking to find the correct item quickly without violating active constraints.

SIFT means:

S - Stateful constraint ledger

I - Information-value question policy

F - Facet-aware routed retrieval

T - Traceable constraint-aware reranking

Inspiration

Shopping intent rarely arrives as one perfect query. A shopper may begin with a broad request, add a budget or use case, say they have no preference for one attribute, and later replace something they said earlier. Conventional search often treats each message in isolation. A conversational system can fail in the opposite direction by asking too many questions, forgetting hard constraints, or sounding helpful while ranking the wrong item.

ShopSIFT was built for a harder formulation: active exact-product retrieval under evolving constraints. The goal is not simply to generate a plausible answer. It is to identify the correct parent product as early as possible, rank it highly, and remain correct when the shopper's intent changes.

What it does

ShopSIFT is a deterministic, multi-turn shopping agent for a frozen 50,000-product catalog. On every turn it can ask one structured clarification question, return ranked recommendations, or do both.

The agent:

  • maintains active, avoided, superseded, unknown, and no-preference states in a structured constraint ledger;
  • lets an explicit intent override replace only the affected preference while preserving unrelated constraints;
  • changes retrieval emphasis according to the evidence available, rather than assuming access to a hidden Buying or Browsing label;
  • combines lexical, category, facet, current-turn, recovery, and catalog-intent routes;
  • asks an allowed attribute only when the expected information value justifies another question;
  • projects visible clues into a catalog-derived intent space;
  • protects exact constraints and high-confidence locks before reranking safer peer candidates; and
  • validates and deduplicates catalog parent_asin values before returning them in scoring order. ShopSIFT also accepts the provided privacy-safe aggregate profile as bounded context, but the shopper's current statements and hard constraints always take precedence.

How it was built

An evidence-gated sequence of experiments was used rather than adding complexity all at once:

  1. E1 - Stateful lexical retrieval. A fast deterministic baseline was established that carries visible conversational evidence across turns.
  2. E2 - Structured constraint ledger. Additions, conflicts, overrides, avoidance, unknowns, and no-preference states were made explicit.
  3. E3 - Adaptive clarification. A value-of-information-inspired policy was added that returns a valid structured ask_attribute while still recommending products.
  4. E4 - Multi-route retrieval and constraint tiers. Different evidence routes improve recall, while tiers stop weaker signals from violating stronger constraints.
  5. E4.5 - Catalog-intent projection. Visible clues are projected into intent representations derived from the same frozen public catalog.
  6. E5 - Guarded ordering. Deterministic semantic evidence reorders only eligible display positions, with exact predecessor fallback when a guard does not pass. The submitted runtime is local and non-LLM. It uses immutable catalog-derived indexes and sidecars, isolated per-session state, deterministic stable tie-breaking, no external API, and no model tokens. Synthetic sessions and later learned-ranker experiments are offline evaluation tools only; they are never used as target-answer data at runtime.

The implementation follows the required Python interface:

class Agent:
    def reset(self, session_id: str, user_profile: dict) -> None:
        ...

    def respond(
        self,
        session_id: str,
        user_message: str,
        turn: int,
        top_k: int,
    ) -> dict:
        ...

The official objective combines retrieval success, rank, and speed of conversion: TechnicalScore = 0.50 × HitRate@10 + 0.30 × MRR + 0.20 × Efficiency

Tools, APIs, assets, and libraries

ShopSIFT was developed in Python with Git and GitHub, and validated with the repository's unittest suite and the unmodified organizer evaluator. The submitted E5 runtime makes no external API or network calls and uses no hosted model.

Its runtime data and assets are the organizer's frozen Amazon Reviews 2023 Clothing_Shoes_and_Jewelry catalog plus the E5 intent-projection sidecar and manifest derived only from that participant-visible catalog. Python dependencies and installation steps are documented. Synthetic sessions and later learned-ranker experiments are offline development assets only and are not runtime sources of target answers.

Results

On the released 200-session public development set, the frozen E5 configuration produced:

System HitRate@10 MRR MTTC TechnicalScore
Official starter 0.125 0.068034 9.810 0.106710
ShopSIFT E5 1.000 0.822490 2.400 0.918747

Relative to E4 on the same public set, E5 kept HitRate@10 at 1.000, improved MRR by 0.024292, improved 10 sessions, tied 190, and recorded no paired regression. Two complete E5 runs were byte-identical.

E5 was also tested on 3,000 product-group-disjoint synthetic sessions. Its TechnicalScore was 0.908304 versus 0.886730 for E4, with all 5 grouped folds and all 3 fixed seeds positive. These sessions intentionally resemble the released deterministic task and are used only for development and promotion decisions.

These are public and synthetic development results, not a claim about the unreleased 800-session final evaluation.

ShopSIFT demonstrated public session

  • Source: released 200-session public development set
  • Public sample: public_0002
  • Scenario: Intent Override
  • Agent configuration: frozen E5 guarded deterministic hybrid
  • Final/private evaluation data used: No
  • Label handling: The released public target was checked only after each Agent response and was never passed to reset() or respond().
  • State annotation: Conversation-state annotation reconstructed from visible user messages; not an additional Agent input or internal debug export.
Turn Visible user message Agent message ask_attribute Top three ranked ASINs Visible state transition
1 I'm looking for Accessories Belts. Buckle closure Do you have a material preference? material B08S6Q2DM2, B078HG9KCY, B00N4CEGEW Active: category: Accessories Belts, Buckle closure; superseded: —
2 For that, what matters is: leather; 100% Leather. Which product feature matters most to you? feature B071X54486, B0C4HGYDMF, B07RQSN383 Active: category: Accessories Belts, Buckle closure, 100% Leather, leather; superseded: —
3 Actually, ignore my earlier preference. What I need is: leather. Do you have a color preference? color B0C4HGYDMF, B08FLYBP2J, B07RQSN383 Active: category: Accessories Belts, 100% Leather, leather; superseded: Buckle closure
  • Outcome: public target first appeared on turn 3 at rank 6.
  • Validation: replayed first_hit_turn and best_rank matched the recorded public-session summary.
  • Recommendation validation: ranking was normalized to catalog-valid, unique ASINs in scoring order.

This demonstration is presentation evidence only. Official metrics come from the unmodified full public evaluator.

Challenges faced

Preserving intent without preserving mistakes

A conversational memory system must retain useful context while forgetting exactly the right thing when the shopper changes their mind. Rebuilding intent from only the latest message lost constraints; accumulating every message created contradictions. The structured ledger gave each value an explicit status and made overrides testable.

Asking a useful question without delaying retrieval

Every clarification consumes a turn, yet a good answer can sharply improve rank. Questioning was treated as a structured action tied to candidate uncertainty, and the agent still returns recommendations in the same turn.

Combining recall and precision safely

Broad routes recover products that a strict query may miss, but an unconstrained fusion can let weak semantic evidence overrule an exact requirement. Constraint tiers, pinned evidence, recovery rules, and deterministic fallback were introduced before reranking was allowed.

Improving the ranker without sacrificing hits

A shallow learned residual ranker was tested after E5. It reduced TechnicalScore by 0.015252 on a fresh group-disjoint synthetic corpus and produced more regressions than improvements, so it failed the promotion gate and was not enabled. That negative result was valuable: model complexity is not progress unless it improves the metric while preserving reliable retrieval behavior.

Matching the evaluator exactly

The simulator responds to the structured ask_attribute, not to a question inferred from prose. Recommendation order is the ranking, optional numeric scores are ignored, and only the first 10 valid unique ASINs are scored. Interface validation and session isolation therefore became part of the retrieval design rather than last-minute packaging work.

What was learned

The strongest lesson was that conversational product search is a joint state, retrieval, ranking, and decision problem. A better text matcher alone is insufficient.

The main takeaways were to:

  • represent changing intent explicitly;
  • align clarification with the evaluator's structured action space;
  • separate high-recall candidate generation from constraint-safe ordering;
  • treat public and synthetic results as development evidence, not private-set guarantees;
  • use negative experiments to protect a strong predecessor; and
  • favor a reproducible local system when added model complexity does not earn its operational cost.

What's next

Future work could improve calibrated question-value estimation, evaluate richer residual ranking on genuinely independent data, and make evidence-based per-product explanations more complete without claiming unavailable variant-level details. The submission deliberately keeps the validated E5 core: deterministic, evaluator-compatible, zero-token, subsecond per-response latency after startup in the cited benchmark, and guarded against unsafe ranking changes.

Built With

  • bm25
  • constraint-tracking
  • conversational-ai
  • deterministic-systems
  • hybrid-retrieval
  • information-retrieval
  • intent-routing
  • natural-language-processing
  • product-search
  • python
  • recommender-system
  • synthetic-evaluation
Share this project:

Updates

Submission history