MG | Devpost

Shopping Copilot — Devpost Written Project Description (TikTok TechJam 2026, Track 4: Shopping Copilot) Inspiration / Problem Statement Most conversational shopping assistants fail in one of two predictable ways: they interrogate the customer with question after question before showing anything useful, or they show generic, one-shot results and never adapt as the customer reveals more about what they actually want. Track 4 asks for an agent that finds a customer's hidden target product within 10 conversational turns — which means every turn has to do double duty: narrow the search and decide whether asking a question is worth the turn it costs. What it does Shopping Copilot is a conversational agent that classifies customer intent (Buying, Browsing, Intent Override, or Boundary) from the opening message, then runs a retrieval strategy suited to that intent instead of forcing every conversation through one generic search: • Buying sessions state a hard requirement up front — the agent immediately treats that requirement as mandatory in the search (not just a ranking nudge), so a customer asking for leather boots never sees a canvas sneaker. • Browsing sessions start broad and get progressively stricter: every preference the customer discloses (color, material, category) is promoted from optional to mandatory in the retrieval query the moment it's mentioned, so the same search function behaves differently turn to turn based purely on how much the conversation has revealed. • Intent Override sessions get a correction that lands on the specific piece of context it applies to, rather than wiping the whole conversation's accumulated state. Throughout, the agent tracks per-attribute state across the whole conversation and asks for at most one missing attribute per turn, skipping anything it can already infer, and stops asking altogether after turn 6 so the tail of the conversation budget goes to pure retrieval. On the 200-session public evaluation set, this took the provided weak baseline's TechnicalScore from 0.1067 to 0.615556 (Hit Rate@10 0.740, MRR 0.4169, mean time-to-hit 4.98 turns). How we built it Development followed one discipline throughout: every change was measured against the real local evaluator on the full public set before being kept, and reverted with the reasoning documented if it didn't help. That discipline is what shaped the final architecture — two significant technical directions were built, measured end-to-end, and deliberately turned off because the numbers said so, not because they seemed promising in theory: • Dense/hybrid semantic retrieval (spaCy word-vector similarity merged with BM25 via reciprocal rank fusion) was built and measured net-negative on this catalog, then fully removed — code and dependency both, not just disabled. • A local LLM re-ranker (Qwen2.5-0.5B-Instruct, quantized to GGUF Q4_K_M, run via llama-cpp-python as a second-stage re-ranker over the BM25 candidate pool) was built and run end-to-end on real consumer hardware. The full 200-session result showed it never surfaced a single new correct answer BM25 hadn't already found, and made ranking (MRR) worse on average by reordering already-correct hits. We then built a full LoRA fine-tuning pipeline — training data generated from real project sessions, a runnable Google Colab notebook, GGUF re-conversion — to teach it a real reranking policy instead of relying on zero-shot prompting. Its own holdout sanity check caught the fine-tune collapsing to a trivial fixed-order prediction before we spent time on a wasted GGUF conversion or full eval run, and we chose to lock in the confirmed-good BM25-only path under deadline pressure rather than keep chasing it. The final, submitted retrieval path is intentionally simple: SQLite's built-in FTS5 full-text index over the 50,000-product catalog, with a hand-tuned field-weight vector across title/categories/features/details/store/description, and the mandatory/optional term-promotion logic described above layered on top in Python. Development tools VS Code (primary local IDE, used by the team for iterating on agent.py and reading evaluator tracebacks), Google Colab (T4 GPU runtime, used exclusively for the LoRA fine-tuning experiment — local hardware had no usable GPU for training), and the terminal-based local evaluator harness (python3 -m evaluator.local_evaluator) provided by the organizer for all scoring. APIs used None. The submitted agent makes no external API calls and requires no network access or credentials — every retrieval and ranking decision runs locally against the provided catalog. This was a deliberate choice: it keeps the submission reproducible under the organizer's option to disable network access for official scoring, and keeps reported token usage at zero because no model call is made on the scored path. Libraries and frameworks used • Python standard library only (sqlite3, re, json, pathlib) for the submitted, scored agent — no third-party runtime dependency at all. • llama-cpp-python — used to build and evaluate the local LLM re-ranker (Qwen2.5-0.5B-Instruct, GGUF/Q4_K_M). Present in the codebase as documented, disabled-by-default infrastructure; not on the scored path since it measured net-negative. • spaCy (en_core_web_md word vectors) and NumPy — used to build and evaluate the dense/hybrid semantic retrieval path. Measured net-negative and fully removed from the codebase (not merely disabled). • Hugging Face transformers, peft, trl, datasets, accelerate, bitsandbytes — used in the Google Colab notebook for the LoRA fine-tuning attempt on the reranker model. • llama.cpp (convert_hf_to_gguf.py, llama-quantize) — used for HF-to-GGUF conversion and quantization in the fine-tuning pipeline. Datasets and assets used • Amazon Reviews 2023 (Clothing, Shoes and Jewelry category), from the McAuley Lab at UCSD — the organizer-provided, frozen 50,000-product catalog (catalog.jsonl) that all retrieval runs against. • 200 labeled public development sessions (data/public_set.jsonl), provided by the organizer, sampled from the official Clothing 5-core leave-last-out split — used for all local development and the reported scores above. The organizer's private 800-session set is not available to us and was not used. • rerank_training_data.jsonl / train.jsonl / holdout.jsonl — a self-generated fine-tuning dataset (169 examples) built by replaying all 200 public sessions through our own agent's BM25 candidate-pool construction, used only for the LoRA fine-tuning experiment described above. Challenges we ran into Getting an honest read on whether the LLM re-ranker actually helped took real hardware time — a full 200-session eval with the local model takes on the order of ten-plus minutes on a CPU-only laptop (Intel i7-1335U, no discrete GPU; the Intel Iris Xe integrated GPU isn't usable by llama.cpp's inference path without a separate build), and it was tempting to reason about pool size and configuration instead of just running the real comparison. We caught ourselves doing that once and corrected it by re-running the actual single-variable test rather than trusting the reasoning. The fine-tuning attempt's failure was itself informative: a 149-example dataset with 61 trivial "already correctly ordered" examples was enough for a 3-epoch LoRA tune to shortcut to always predicting the identity order, which we only caught because we built a holdout sanity check before spending time on GGUF conversion. Accomplishments that we're proud of Nearly 6x'ing the baseline TechnicalScore (0.1067 → 0.615556) using a fully offline, zero-dependency, zero-cost retrieval pipeline — and having a real, measured, documented reason for every architectural decision along the way, including the ones we reverted. What we learned The two "obviously should help" upgrades — semantic retrieval and an LLM re-ranker — both measured as net-negative on this specific catalog and session distribution, while several unglamorous fixes (matching the evaluator's own disclosed-value regex exactly, tuning BM25 field weights, promoting disclosed terms from optional to mandatory) accounted for most of the real score improvement. Measuring against the real evaluator before keeping any change caught this early instead of after final submission. What's next The two directions still open, evaluated but not completed: a rebalanced LoRA fine-tune (downsampling the trivial identity-order training examples so the model can't shortcut past learning a real reranking policy) and scenario-specific field-weight tuning to recover the small Intent Override hit-rate dip introduced by the final BM25 weight sweep.