sideswipe

What it does

sideswipe carries on a multi-turn conversation with a customer and returns a ranked top-10 list of products from a 50,000-item catalog. Each turn it parses the customer's free-text reply into structured constraints (material, color, size, style, budget, use case, brand, or a catch-all feature bucket), asks one clarifying question, and re-ranks candidates through a multi-field lexical index, a locally-trained linear ranker, and a statistical semantic reranker. It closes most sessions in 2-4 turns by always asking a general "other" question instead of a named attribute — a strategy tuned to how the evaluator's simulated customer discloses intent-card details. No LLM, external API, or network call is involved anywhere in the pipeline.

How we built it

Everything runs on the Python standard library — no LLM, no external API, no network access, and no third-party runtime dependencies (model training used scikit-learn offline; the runtime only needs stdlib, including an in-memory sqlite3 FTS-style index). The pipeline is layered: SessionState (shopping/conversation.py) tracks slots and exclusions and parses replies; CatalogIndex (shopping/retrieval.py) does multi-field lexical retrieval; a LinearRanker + FieldSignalRanker (shopping/ranker.py), trained offline on the public session set, scores candidates; a SemanticReranker (shopping/semantic_reranker.py) blends in a small local TF-IDF-style similarity model; and a lightweight buyer-state FSM (shopping/buyer_state.py) tunes how aggressively the candidate pool widens on low confidence. We merged two previously separate prototypes — one built around the retrieval/ranking pipeline, the other around the clarification policy — into this single agent, and validated every change against both the official local_evaluator on the 200-session public set and our own overnight synthetic sweeps (1,000+ catalog products, 2,000+ sessions per scenario pair).

Challenges we ran into

Composing two independently-built halves — the retrieval/ranking pipeline and the always-ask-other clarification policy — without either one regressing the other's score.
The Boundary scenario's one-time refusal ("I don't have a preference for X; please use your judgment") looks identical to genuine exhaustion at first glance. Conflating the two silently broke the clarification policy on that scenario until we split detection into two separate regular expressions.
The whole other-first trick depends on a specific quirk in the evaluator's customer_reply() (2 disclosed items per "other" question vs. 1 for a specific attribute). Confirming the gain was real, and not an artifact of the small 200-session public set, required building and running large synthetic scenario sweeps overnight.
Tuning ranker weights, diversity caps, and backfill thresholds against only 200 public sessions without overfitting to that small sample.

Accomplishments that we're proud of

0.9201 technical score and 0.995 Hit Rate@10 on the 200-session public set, fully offline and at $0 marginal cost per session.
Confirmed the gain wasn't a small-sample fluke: at synthetic scale (1,000 catalog products, 2,000+ sessions per scenario) our agent beat the prior adaptive-questioning baseline on every scenario, with the largest jump on Boundary (0.730 → 0.836).
A genuinely LLM-free agent. Every scoring decision traces back to a local, inspectable weight file, with zero network dependency and reproducible latency (~1-2 minutes for the full public set on one CPU core).
Being upfront in our own writeup about where the strategy is a legitimate improvement (the retrieval/ranking stack) versus where it's exploiting a quirk of the simulator (the other-first clarification policy), rather than overselling it.

What we learned

Small evaluation sets can hide whether a strategy is a genuine improvement or a small-sample artifact. The synthetic scale run is what actually gave us confidence in the other-first policy.
A policy can be locally optimal against a simulator's specific reply logic while still reading as a worse user experience: the other-first strategy scores well precisely because it never adapts to what the customer actually says, which pushed us to separate "wins on this benchmark" from "the right behavior for a real shopping assistant."
Small, careful parsing fixes can matter as much as the model architecture. The Boundary scenario's biggest score gain came from distinguishing one-time refusal from permanent exhaustion, not from a smarter ranker.
Splitting the system into narrow, independently testable pieces (conversation state, retrieval, ranking, semantic reranking, buyer-state FSM) let two people build and validate different halves of the pipeline in parallel and merge them without conflict.