ShopSense (for Tiktok Techjam)

Not just keyword matching, it's a shopping assistant that understands intent, remembers context, and asks the right question at the right time

Comment

Inspiration

Traditional e-commerce search engines rely on static keyword matching, which struggles to distinguish between a shopper who knows exactly what they want and one who is casually exploring — and it has no way to understand that "sneakers" and "running shoes" mean roughly the same thing. I wanted to build a shopping assistant that actually converses with a shopper: remembering what's been said, asking the right question at the right time, and understanding meaning rather than just matching words.

What it does

CartTalk conducts a multi-turn conversation with a shopper and returns ranked product recommendations. On each turn, it:

  • Detects whether the shopper is in "buying" mode (specific, ready to purchase) or "browsing" mode (open-ended, exploring)
  • Retrieves candidate products using a hybrid of keyword search (BM25) and semantic/embedding search, weighted according to the detected intent
  • Tracks accumulated shopper preferences (category, material, color, size, budget, etc.) across the whole conversation, correctly handling both incremental detail ("size 9" after "running shoes") and abrupt changes of mind ("actually, forget shoes — show me jackets")
  • Asks a targeted clarifying question when the candidate pool is too broad to guess confidently, but is capped so it never stalls the conversation
  • Uses an LLM (Claude Haiku) to do a final re-ranking pass over the top candidates and generate a short summary of the shopper's inferred preferences, which is carried forward into later turns

How we built it

I started from the organizer's weak BM25 baseline and the local evaluator, and built up in stages, testing against the evaluator after every change:

  1. Conversation state first. Before touching retrieval quality, I built per-session slot tracking, override detection, and clarification logic on top of the existing BM25 baseline. This alone took the technical score from 0.107 to roughly 0.295 — mostly by fixing catastrophic failures on vague ("browsing") and edge-case ("boundary") queries that keyword search alone could never handle.
  2. Hybrid retrieval. I added dense embedding search (sentence-transformers) and combined it with BM25. I initially planned to always blend both signals, weighted by intent — but empirical testing showed pure BM25 actually outperformed any blend for "buying"-intent queries, so I adopted a data-driven split: BM25-only for buying, after testing confirmed embeddings measurably hurt precision there; a 40/60 BM25/embedding blend for browsing.
  3. LLM ranking. I added a final Claude Haiku ranking pass over the retrieved shortlist, along with a preference-summary step that feeds forward into subsequent turns' ranking calls (my "self-evolution" / dynamic context piece).
  4. Iteration and tuning, informed directly by reading the evaluator's own scoring code — I extracted the exact ALLOWED_ATTRIBUTES set and constraint-classification logic from local_evaluator.py so my clarifying questions would always be ones the simulated shopper could actually answer, and derived my clarification-budget size from the scoring formula itself (since a total miss scores worse than the slowest possible success, I deliberately favored accuracy over speed).

Challenges we ran into

  • A local-LLM speed wall. I initially built the ranking step against a local Ollama model to avoid any API cost, but a full 200-session evaluation run (up to ~2,000 sequential LLM calls) took well over an hour on a laptop — impractical for iteration and risky for a judged run against 800 private sessions. I switched to a cheap, fast hosted API (Claude Haiku) instead, which cut evaluation time from over an hour to a few minutes. I also hardened the evaluation workflow itself: a --limit flag on the evaluator for fast smoke-testing on a subset of sessions, live progress logging (Evaluating session 12/200 (sample_id)...) so a run's status is never ambiguous, and confirmed the LLM ranking call is skipped entirely on turns where the agent only returns a clarifying question — since ranking is irrelevant until a final recommendation is made.

  • Small-model JSON reliability. Getting a fast model to reliably return valid JSON took real debugging — I had to handle extra trailing braces, multi-JSON-object responses (the model second-guessing itself mid-answer), and text wrapped around the JSON, eventually settling on an assistant-message prefill technique plus a brace-matching parser that extracts exactly the first complete JSON object.

  • A subtle non-determinism bug. My results were silently non-reproducible for a while — identical code producing different scores on every run — traced back to using a Python set (which has randomized iteration order) for my list of trackable attributes, affecting which clarifying question got asked first each run. Switching to a tuple fixed it completely.

  • A prompt-engineering attempt that backfired. I tried giving the LLM more detailed ranking instructions and additional context, expecting an improvement — it measurably regressed performance instead, and I reverted to the simpler prompt. A useful reminder that more instructions isn't automatically better for smaller/faster models.

Accomplishments that we're proud of

Every metric, across every one of the four evaluation scenario types, improved over baseline with zero regressions — including "boundary" cases going from complete failure (0.0 hit rate) to functioning, and "browsing" scenarios improving roughly 16x on hit rate. Overall technical score improved from 0.107 to 0.299, essentially a 2.8x improvement, achieved through a genuinely complete implementation of all four required architectural pillars.

What we learned

That empirical testing should override initial architectural intuition — my decision to use strict BM25-only retrieval for buying-intent queries came directly from data showing my original "always blend" design was actually hurting precision there. I also learned firsthand why local LLMs, while free, come with real practical tradeoffs in speed that matter a lot for an evaluation harness with hundreds of sessions.

What's next for CartTalk (for Tiktok Techjam)

  • Expanding slot-extraction vocabulary beyond fixed word lists (e.g., using the LLM itself to extract structured constraints, rather than regex-based keyword matching)
  • Systematic hyperparameter tuning of blend weights and over-generality thresholds, rather than single-pass manual testing
  • Investigating why buying-intent queries — despite having more explicit shopper detail — currently score lower than browsing-intent queries, likely tied to the extraction-vocabulary gap above

Development tools

Cursor (editor + terminal), GitHub Desktop (version control)

APIs used

Anthropic Claude Haiku (claude-haiku-4-5-20251001) for LLM-based semantic ranking and preference summarization

Libraries and frameworks used

sentence-transformers (dense embedding retrieval, all-MiniLM-L6-v2 model), sqlite3 (BM25 full-text search via FTS5), anthropic (Python SDK), numpy

Datasets and assets used

TechJam 2026 frozen catalog (50,000 products, Amazon Reviews 2023 Clothing/Shoes/Jewelry category) and the organizer's 200-sample public development set, both provided via the official participant kit.

Built With

Share this project:

Updates