tiktoktechjam_shopping_copilot_khaitanasmita

tiktoktechjam - shopping copilot

Comment

Inspiration

Shoppers almost never start with a perfect query. They browse (“something for winter”), they already know one hard constraint (“must be leather”), or they change their mind mid-conversation (“ignore that — I need water resistant”). That is also how people shop on TikTok Shop: the catalog is huge, the first message is messy, and the product they would actually buy is hiding behind a few missing attributes.

We chose the Conversational E-Commerce Search track because the metric is honest. The hidden target is a real purchase from Amazon Reviews 2023. You either put that parent_asin in the top-10 or you do not. The official weak baseline scores $\text{TechnicalScore} = 0.107$ on the 200-session public set. We wanted an agent that is accurate, cheap, and judge-reproducible — including when the organizer turns the network off.

That constraint became the brief. Build a hybrid retrieval and dialog system first. Treat any live LLM as optional.

What it does

Shopping Copilot is a multi-turn agent. A simulated shopper talks in natural language. The agent asks one structured clarifying question per turn and returns a ranked top-10 of catalog IDs from a frozen $50{,}000$-product Clothing, Shoes & Jewelry catalog.

On every turn it classifies intent (buying / browsing / unknown), updates slot-aware state, retrieves from three tracks in parallel, fuses the lists, and asks the next most useful attribute:

$$ \texttt{feature} \rightarrow \texttt{material} \rightarrow \texttt{color} \rightarrow \texttt{use_case} \rightarrow \texttt{style} \rightarrow \texttt{size} \rightarrow \texttt{budget} \rightarrow \texttt{other} $$

It covers all four official scenarios: buying (hard constraint on turn 1), browsing (vague start), intent override (replace the old preference, do not append it), and boundary (no preference for the asked attribute).

The organizer scores

$$ \begin{aligned} \mathrm{HR@10} &= \frac{#{\text{sessions with target in top-10}}}{N}, \ \mathrm{MRR} &= \frac{1}{N}\sum_{i=1}^{N} \frac{1}{\mathrm{rank}i}\quad (\text{miss}=0), \ \mathrm{MTTC} &= \frac{1}{N}\sum{i=1}^{N} t_i\quad (t_i=11\text{ on a miss}), \ \mathrm{Efficiency} &= \mathrm{clip}!\left(\frac{11-\mathrm{MTTC}}{10},\,0,\,1\right), \ \mathrm{TechnicalScore} &= 0.50\,\mathrm{HR@10} + 0.30\,\mathrm{MRR} + 0.20\,\mathrm{Efficiency}. \end{aligned} $$

On the public set, with no API key: $\mathrm{HR@10}=0.940$, $\mathrm{MRR}=0.692$, $\mathrm{MTTC}=3.73$, $\mathrm{TechnicalScore}=0.823$. By scenario, Hit@10 is $0.950$ buying, $0.913$ browsing, $0.967$ intent-override, $1.000$ boundary.

One buying session, hidden target GLOBALWIN Women's Waterproof Winter Boots:

  1. Shopper: “I'm looking for Outdoor & Work Snow & Cold Weather. A key requirement is: leather.” The agent skips asking material and asks feature.
  2. Shopper discloses rubber sole and shaft height. Phrase match; ask color.
  3. Shopper has no color preference. Ignore the noise; ask use case.
  4. Shopper discloses Thermolite insulation rated to $-15^\circ\mathrm{F}$ and faux-fur lining. Target is rank $1$ of $10$. Hit.

The same loop finds a plus-size tunic while browsing (rank $3$ on turn $3$) and a Casio watch after an intent override (rank $1$ on turn $3$).

How we built it

The agent is one Python module, starter/agent.py, on the official evaluator. No server, no GPU, no custom training.

Retrieval. SQLite FTS5 runs field-weighted BM25, with title $\times 6$, categories $\times 4$, features $\times 2.5$. A disclosed constraint becomes an exact phrase query; an AND sub-query promotes products that match every disclosed term. A local MiniLM-L6-v2 index ($384$-d, cached after the first run) covers vague language. sklearn TF-IDF with $40\mathrm{k}$ unigram+bigram features adds diversity and is the dense fallback.

The three ranked lists are fused with weighted Reciprocal Rank Fusion ($k=60$):

$$ \mathrm{score}(d)=\sum_{i\in{\mathrm{bm25},\,\mathrm{dense},\,\mathrm{tfidf}}} \frac{w_i}{k+\mathrm{rank}_i(d)+1} $$

Intent sets the blend. Buying is BM25-heavy $(w_{\mathrm{bm25}},w_{\mathrm{dense}},w_{\mathrm{tfidf}})=(0.65,0.10,0.25)$. Browsing shifts mass to the semantic tracks $(0.45,0.20,0.35)$. Then a 10% popularity tie-break, using the fact that every catalog row has ratings:

$$ \begin{aligned} \mathrm{pop}(d) &= \mathrm{rating}(d)\cdot\log\bigl(1+\mathrm{n_reviews}(d)\bigr), \ \mathrm{final}(d) &= 0.90\cdot \widetilde{\mathrm{rrf}}(d) + 0.10\cdot \widetilde{\mathrm{pop}}(d). \end{aligned} $$

That 10% term lifted MRR by about $9.2\%$ with no API cost.

Dialog. Each session keeps isolated buckets for profile tags, category, first-turn terms, disclosure phrases, and overrides. If turn 1 already said “leather,” we pre-mark the material slot. On “ignore my earlier preference,” we drop stale first-turn terms, keep product disclosures, and switch to buying. We never ask brand: the official classify_constraint() never returns it.

Optional LLM. DeepSeek deepseek-chat re-ranks the top-$30$ at most once per session, and only after a new hard constraint appears. Missing key or failed call $\rightarrow$ RRF order. Official scoring is $0.823$ fully offline.

Development tools. Python 3.10+, SQLite FTS5, pytest, a laptop CPU.

APIs. DeepSeek Chat (optional, OpenAI-compatible client). No other external APIs. Official scoring does not need a key.

Assets. Official participant kit: catalog.jsonl ($50\mathrm{k}$ products) and public_set.jsonl ($200$ sessions), from Amazon Reviews 2023 (McAuley Lab, UCSD; Clothing, Shoes & Jewelry). MiniLM-L6-v2 weights (Apache 2.0) download once into the Hugging Face cache.

Libraries. scikit-learn, numpy, sentence-transformers, openai, pytest.

Reproduce: clone the repo, download catalog.jsonl, pip install -r requirements.txt, run python3 -m evaluator.local_evaluator. Offline latency is about $5$–$15\,\mathrm{ms}$ per turn. Optional DeepSeek adds $\sim 1\,\mathrm{s}$ and about $\$0.01$–$0.05$ for $200$ sessions.

Challenges we ran into

The ideas that looked most “AI” were the ones that hurt.

LLM query expansion on browsing turn 1 rewrote “beach trip” into “swimwear sandals.” When the hidden target was something else, BM25 and MiniLM chased the wrong category. Score fell from $0.823$ to $0.784$. We deleted that path.

Raising browsing dense weight to $0.55$ drowned phrase precision. $\mathrm{TechnicalScore}$ fell to $0.681$. Disclosed constraints on this track are exact catalog text, so BM25 has to stay in charge.

Asking the wrong attribute wastes turns. Putting material first, or including brand, inflated buying MTTC. feature has to go first: it is the catch-all the evaluator actually fills.

And judges may disable the network. The scored path therefore uses no API key. MiniLM downloads once, then runs offline.

Accomplishments that we're proud of

We moved $\mathrm{TechnicalScore}$ from the official weak baseline of $0.107$ to $0.823$ with no paid API — a $7.7\times$ gain on a laptop CPU.

$94\%$ of $200$ public sessions put the hidden purchase in the top-10, in $3.73$ turns on average. Intent-override Hit@10 is $0.967$; boundary is $1.000$.

A held-out $40$-session test split scored higher than train ($0.839$ vs $0.795$). The agent uses structural rules — phrase matching, slot filling, replace-not-append — not memorized ASINs.

The whole agent is one Python file. pytest tests/ -v is $19$ passed. After the dense index is cached, a full public-set eval is about $50$ seconds.

We are equally proud of what we did not ship. Query expansion and a dense-heavy blend both looked sophisticated. Both lost to the simpler hybrid.

What we learned

Conversational shopping on this catalog is mostly state plus phrase matching, not a bigger language model. The simulator reveals real product text. The winning move is to ask the attribute that unlocks that text, then phrase-match it.

Neural retrieval still earns its keep on vague browsing, but only as a coverage track with $w_{\mathrm{dense}}\in[0.10,0.20]$. Popularity is a cheap MRR tie-break if you keep it at $10\%$. An LLM that guesses the product type is a liability.

The other lesson was process. Every idea — including the ones that failed — went through the official evaluator. That is how we found out that “smarter” reranking can be a $0.14$ TechnicalScore regression. Measuring beat intuition.

What's next for tiktoktechjam_shopping_copilot_khaitanasmita

Replace MiniLM with a stronger local ranker (a larger bi-encoder or a late-interaction model) so browsing is less dependent on term overlap.

Learn the RRF weights $w_i$ from session trajectories instead of using a fixed buying/browsing blend.

Add cross-session preference embeddings so a returning shopper cold-starts with better priors — closer to how TikTok Shop personalization would work in production.

Keep the offline fallback. Anything that needs a live API stays optional.

Built With

Share this project:

Updates

Submission history