TrippyShoppy

TrippyShoppy is a deterministic, offline-first conversational shopping agent built for the TechJam 2026 conversational-search challenge. It asks one sharp clarifying question on the same turn it returns up to ten ranked products, remembers everything the shopper discloses, and never re-shows a product it has already offered. Its headline configuration, O+, scores a TechnicalScore of 0.9189 on the 120-session stratified development split and 0.9097 on the 80-session holdout — every point of lift over the previous build coming from better ranking, with hit rate held flat.

It runs on CPU, calls no hosted model or paid API, and reports zero prompt and completion tokens.

The organizer drives it through two calls — Agent.reset(session_id, user_profile) once, then Agent.respond(session_id, user_message, turn, top_k) each turn. Every response carries customer-facing message text, exactly one allowed ask_attribute (or null), an ordered recommendations list, and zeroed usage. A session ends on the first valid hit, or after turn 10.

Why we built it this way

The scoring protocol rewards two things: finding the hidden product early, and placing it high in the returned list. Three consequences shaped the whole design:

Clarification is free within a turn. A question ships alongside the recommendations rather than replacing them, so staying silent doesn't speed anything up — it just throws away a chance to gather evidence for the next turn. So TrippyShoppy always asks something useful.
Filters are too brittle for this catalog. Metadata is sparse and inconsistent; a hard filter on a mis-parsed constraint can delete the true target. So constraints are recoverable penalties, never filters — a wrong guess can demote the target but never remove it.
Ranking stays conservative. Every reordering stage preserves candidate membership unless a mechanism has a specific, defensible reason to change it. Recall is earned once, in retrieval, and protected everywhere after.

Design choices worth calling out

Exclusion filter (no-repeat recommendations). Within a session we withhold any product already shown and scored. Two motivations. First, interaction design: in a fast, multi-turn exchange, re-surfacing items the shopper just scrolled past wastes a slot and quietly erodes trust — every refreshed result should be something new. Second, it's provably free: the evaluator scores every returned ASIN and ends the session on a hit, so a product that survived a prior turn is, by definition, not the target — dropping it costs zero recall while sharpening the next list. We treat this as the natural seam for richer product logic: in a real deployment this is where we'd layer a recycling policy — time-based decay so an item can resurface after a cooldown, or user-level preference signals that re-admit a product the shopper has since warmed to — instead of a hard per-session block. A safety valve keeps it recall-safe today: if withholding would empty the pool, we return the unfiltered set rather than nothing.
Fitted weights, not guessed ones (O+). O's scoring magnitudes were hand-picked. O+ replaces the eight most load-bearing ones — the fused-score multiplier, the precision-route lexical/hybrid split, the constraint match bonus, the per-attribute hard penalties, and the soft-preference decay and floor — with values learned by black-box search (random search then Nelder-Mead), then frozen. The payoff is concentrated in MRR (0.821 → 0.871 over the full set) with HitRate@10 unchanged.
Recoverable constraints. Hard constraints apply bounded penalties and one bounded component per attribute, so two values in one coarse bucket can't stack into a runaway score. An empty detailed query relaxes to its category before any global fallback.
Membership-preserving reranking. The popularity prior is a small, log-bounded rating-count nudge applied inside the frozen Top-K. It can reorder but never add or remove a product — relevance always outranks popularity.
Fail safe, always. Duplicate or out-of-order turns get the last valid recommendations; any unexpected exception falls back to the last non-empty result or a deterministic catalog list, and still returns a contract-valid shape.

How it works

Verify local assets. The loader reads the immutable 50,000-product JSONL catalog and checks the official file against its pinned SHA-256 digest.
Parse and remember. A deterministic parser extracts category, hard constraints, soft preferences, declines, and intent changes into session state. New disclosures accumulate; a genuine override retires only the earliest superseded soft preference and leaves unrelated constraints intact.
Build the live query. Only active slots contribute. Previously returned ASINs ride along as session exclusions — and an intent override clears that list, because those products were offered against the old intent.
Retrieve locally. Weighted SQLite FTS5 BM25 and a vendored all-MiniLM-L6-v2 encoder produce candidate lists fused by reciprocal rank. Buying turns use a lexical-weighted union for precision; other turns use balanced hybrid fusion.
Score without destructive filters. Per-attribute constraint evidence and intent-aware weights adjust candidate scores; hard constraints are bounded penalties, never filters.
Rank the response. O+ freezes Top-K membership, orders candidates by which active disclosures they satisfy — under the fitted fusion/constraint balance — then applies the log-bounded rating-count prior inside that same set.
Ask the next useful question. The policy scores live candidate facets by normalized information gain, skips declined attributes, and asks a targeted or open clarification alongside the recommendations.
Fail safely. No turn ever returns an invalid or empty response.

Submission configuration

An unset TRIPPYSHOPPY_CONFIG selects the hardened default; an unknown value falls back to the standard-library baseline A. The accuracy path — shared by O and its fitted refinement O+ — enables hybrid retrieval, session memory, bounded constraint scoring, dynamic intent weights, information-gain clarification, no-repeat recommendations, disclosure-order reranking, and the bounded popularity prior. O+ adds only the eight fitted weights on top; because each fitted field's default reproduces O's shipped magnitude, selecting O reproduces the original behavior exactly.

Dense retrieval is optional at runtime: if NumPy, Sentence Transformers, or the verified local model can't load, the retriever degrades to deterministic BM25 with no network request and no failed turn. No LLM ranker or cross-encoder is enabled.

Tools, libraries, APIs, and cost

Python 3.10+; reportable evidence used CPython 3.12.13.
SQLite FTS5 for weighted lexical retrieval.
NumPy, PyTorch, Transformers, Tokenizers, Sentence Transformers for the optional local dense path.
Vendored all-MiniLM-L6-v2 weights, loaded by local path on CPU with local_files_only=True.
pytest across unit, integration, policy, retrieval, evidence, and documentation checks.
No API key, hosted model, external vector database, or paid service.
Prompt tokens 0, completion tokens 0, estimated model/API cost $0.

Data and privacy

The frozen catalog holds 50,000 Clothing, Shoes and Jewelry products; the public development set holds 200 labeled sessions (80 Buying, 80 Browsing, 30 Intent Override, 10 Boundary). The organizer keeps 800 private sessions for final scoring, and only catalog-valid parent_asin identifiers are scored.

The data derives from Amazon Reviews 2023 (McAuley Lab, UCSD). Direct identifiers, timestamps, free-text reviews, raw purchase histories, hidden intent cards, and simulator internals are excluded from the participant data. TrippyShoppy sees only safe aggregate profile fields and the current session's messages; it performs no cross-session profiling and never mutates the catalog. Full provenance and redistribution terms live in DATA_ATTRIBUTION.md.

Evaluation

The evaluator reports HitRate@10, mean reciprocal rank (MRR), and mean turns to conversion (MTTC), combined as:

TechnicalScore = 0.50 × HR@10 + 0.30 × MRR + 0.20 × efficiency

where lower MTTC raises efficiency. TechnicalScore is evidence of technical execution, not the competition's entire judging decision.

The lineage below lists only the builds that materially moved accuracy:

Config	Accuracy-relevant change	Dev	Holdout
`P`	Hybrid retrieval, state, constraints, dynamic routing, frozen-set phrase reranking	`0.819939`	`0.843958`
`Q`	`P` plus a bounded rating-count prior	`0.862083`	`0.880321`
`O`	No-repeat recommendations plus disclosure-order ranking on the `Q` branch	`0.908416`	`0.898795`
`O+`	`O` with its eight scoring weights fitted rather than guessed	`0.9189`	`0.9097`

[Over the 200 public sessions] O+'s gain over O is primarily in ordering: HitRate@10 is unchanged at 0.9850, MRR rises from 0.8211 to 0.8708 across all 200 sessions, and the improvement holds on the holdout (+0.0138).

Reportable evidence. Configuration O+ produced clean, reportable CPU runs with an immutable input snapshot, the pinned model and dependency lock, true hybrid retrieval, and zero agent exceptions, evaluator exceptions, or invalid responses:

Split	Sessions	HR@10	MRR	MTTC	TechnicalScore
Dev	120	`0.983333`	`0.88869`	`2.96667`	`0.91894`
Holdout	80	`0.9875`	`0.854286`	`2.875`	`0.912536` (exploratory)

The dev row is recorded at f3731ea1, the holdout row at f3731ea1. O+'s figures are measured locally on the same official split; a reportable run on the reference platform is the remaining step before promotion.

The 120/80 split is a project-imposed control over the 200 public sessions, not an organizer-defined hidden test. Holdout results for Q, O, and O+ are labeled exploratory because the popularity hypothesis used aggregate public target evidence and these configs were selected through development-split isolation. We therefore present the mechanism and each metric, not only the composite score.

Runtime and reproducibility

The reportable runs used Linux x86-64 under WSL2, SQLite 3.46.1, CPU-only inference, and the hash-pinned requirements-dense.lock.txt environment with zero lock mismatches.

Measurement	Dev	Holdout
Turn latency p50 / p95	`93.6 / 145.8 ms`	`107.9 / 168.1 ms`
Turn latency mean / max	`98.4 / 175.3 ms`	`113.6 / 311.8 ms`
Peak RSS	`1.95 GB`	`1.98 GB`
One-time reportable initialization	`1307.1 s`	`1274.7 s`

The long initialization deliberately rebuilds all 50,000 catalog embeddings in-process so the evidence can bind their provenance. Normal use reuses a fingerprinted local embedding cache and starts in seconds. Setup, evaluation commands, hashes, and evidence rules are in the repository README.

Limitations

The parser targets the organizer's controlled language and common commerce terms — it is not a general NLU model.
Catalog metadata is sparse: color is missing from more than half of products, so color evidence is weaker than better-populated fields.
The rating-count prior can favor established products over niche or newly listed ones, and public target construction may amplify that bias.
No-repeat recommendations are recall-safe under this evaluator because every returned ASIN is scored and a hit ends the session; a continued real-world conversation wouldn't prove every earlier recommendation was wrong (hence the recycling extensions above).
Boundary is the smallest scenario bucket, so its per-scenario metrics are directional rather than stable.
The reportable dense path has a large CPU cold start (~2 GB peak), though cached normal use is far faster.
TrippyShoppy has no image input, external vector database, full-model training, hosted-LLM dependency, or real transaction support.

Research credit

The retained policy uses information gain, not the experimental EVPI mode. Config U independently adapted the EVPI framing as a deterministic, target-free expected Top-K utility; its clean dev run at 87834f4 held HR@10 at 0.941667 and raised MRR to 0.641323, but MTTC rose to 3.175000 and TechnicalScore fell to 0.819730, below the pre-registered 0.819939 gate, so U was rejected without opening holdout.

Sudha Rao and Hal Daumé III. 2018. Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information. Proceedings of ACL 2018, pages 2737–2746.

DOI: 10.18653/v1/P18-1255 · ACL Anthology. Licensed CC BY 4.0. TrippyShoppy did not reproduce or port its neural model, code, training data, annotations, or weights; the full adoption boundary is in research-attribution.md.

Preference-state and clarification work was also informed by Li et al.'s TRACER method in Wizard of Shopping (ACL 2025), and clarification-quality guards and rerank reporting by Ye et al.'s ProductAgent (arXiv:2407.00942) — both implemented independently. The repository contains no Wizard of Shopping records, ProductAgent code, or AliMe KG data; see the Wizard of Shopping audit and ProductAgent audit.

Team contributions

Git history is the source of truth for the identities below.

Repository identity	Contribution visible in history
Danixjg	Initial deterministic clarification, stateful BM25 policy, override handling, integration, and repository maintenance
Kivye / kivye	Hybrid retrieval, evidence hardening, state and clarification improvements, phrase/popularity ranking, research integration, and review
MaxLZE	Reproducibility gates, retrieval/ranking ablations, source audits, documentation integrity, and promotion of configuration `O`
suwi1226	No-repeat configuration `N`, disclosure-order configuration `O`, and learnt-weights configuration `O+`, error analysis pipeline, retrieval/reranking ablations
pranavpillaiNUS	Configuration `O` integration, ProductAgent audit integration, and turn-by-turn error analysis
thaqifrafe	Clarification-timing diagnostics, configurations K and L