• 

  AgentShopper's Architecture

• 

  AgentShopper's Technical Score Climb Overtime

• 

  AgentShopper's Headline Metrics

About the project

Inspiration

Most "smart search" demos assume the shopper says exactly what they want on turn one. Real shopping isn't like that — you say "something for a wedding," get shown options, realize you actually meant flats not heels, and maybe change your mind about color halfway through. The TechJam 2026 "Shopping Copilot" brief was explicit about this: intent routing, multi-turn evolution, context programming, and evaluation, scored end-to-end. That combination — a system that has to converge on one specific product across up to 10 turns, against a real 50,000-item catalog, with a hard metric at the end — was the appeal. Not a chatbot that sounds good, one that's graded on whether it actually finds the thing.

What it does

Agent Shopper is a conversational agent that narrows a 50k-item Amazon Clothing_Shoes_and_Jewelry catalog down to one target product through natural-language dialogue. It routes each turn along a buying/browsing axis, retrieves via three fused routes (BM25, TF-IDF/dense vectors, category filters), reranks candidates through a heuristic-or-cross-encoder stage, and tracks slots (category, material, color, size, budget, …) across turns — including detecting when a shopper reverses an earlier preference and rewriting state accordingly rather than blindly accumulating contradictory constraints.

Against the organizer's own evaluator on the 200-session public dev set:

How we built it

The retrieval core fuses three independently-scored routes with weighted Reciprocal Rank Fusion:

$$ \text{RRF}(d) = \sum_{r \in \text{routes}} w_r \cdot \frac{1}{k + \text{rank}_r(d)} $$

On top of that sits a SessionState that accumulates slots turn by turn and distinguishes plain information-gathering from an intent override (explicit contradiction language, a conflicting category, a non-overlapping budget) — which clears and reopens only the affected slots rather than nuking the whole state. A lightweight orchestrator then decides, per turn, whether to retrieve, ask a clarifying question, or relax an over-constrained filter, based on turn budget and pool size.

The part we're most glad we did: nothing shipped as a default without an A/B run through the real evaluator first, and anything with enough free parameters to overfit 200 sessions (the reranker weights, the override-probability model) went through stratified k-fold cross-validation, not a single train/eval split. Of 21 tuning attempts logged, close to half regressed or came back neutral — and those were reverted completely, dead code and all, rather than shipped behind a disabled flag. The one exception worth calling out: adding a frozen local cross-encoder reranker, which went through offline replay, CV, a fault-injected circuit-breaker test, and a full re-evaluation against the extracted submission archive — not the working tree — before we trusted it as the default.

Challenges we faced

The one that cost the most time wasn't algorithmic, it was infrastructure: the cross-encoder checkpoint we originally intended to use (ms-marco-MiniLM-L-6-v2) reliably crashed the process with a SIGBUS during inference — an OS-level signal, not a catchable Python exception, so no amount of try/except could paper over it. Root-causing it meant bisecting across model sizes and architectures until we found a checkpoint (ms-marco-TinyBERT-L-6) that didn't crash, then building a subprocess-isolated smoke test so a native crash could never again take down a whole evaluation run silently.

A quieter but more dangerous bug: our submission-archive builder's file allowlist didn't include the dense retrieval model directory. DenseIndex degrades gracefully when its model is missing — meaning a judge's run would have silently lost an entire retrieval route with no error at all, just a quiet drop in Hit Rate@10 against every number in this README. We only caught it because we'd built the habit of extracting each submission archive into a fresh directory and re-running the full evaluation against that, never the working tree, before trusting anything as final.

And one honest, still-open gap: 26 of 200 public sessions never recall the target from any route at any depth, no matter what happens downstream. Diagnosing it (rather than guessing) meant writing a dedicated per-feature contribution script, which showed the bottleneck is upstream of reranking entirely — a rating-based candidate-injection step that never gives most of these targets a shot at being considered in the first place.

What we learned

The biggest lesson was procedural, not technical: a negative result you keep around as a disabled toggle is worse than no experiment at all — it's a landmine for a future version of yourself who assumes dead code is load-bearing. Reverting cleanly, every time, made the codebase trustworthy enough that we could keep iterating fast without re-litigating old decisions.

The second: for a system this stateful, the eval harness is the spec. A hand-picked weight or a plausible-sounding heuristic ("hard-filter on explicit requirement language") can look obviously correct and still measure worse than doing nothing — the turn-1 hard-constraint regex that accidentally matched the simulator's own scripted opening line, and quietly tanked buying-scenario Hit Rate@10 from 0.51 to 0.41, was a reminder that plausible and validated are different claims, and only one of them belongs in a shipped default.

Appendix: the full ablation log

Every one of these was measured with an A/B (or, where noted, a stratified k-fold CV) run of the official local evaluator on the 200-session public dev set — never eyeballed. "Shipped" means it's the current default in config.py; everything else, including a mechanically-confirmed-correct change, was reverted in full rather than kept as a disabled toggle.

Summary

For more details look at the GitHub's README.md and doc/experiments.md

Built With

• numpy
• pydantic
• python
• pytorch
• scikit-learn
• sentencetransformer