ShopPilot

Architecture Diagram

Overview

Shopping Copilot is a conversational shopping agent that finds the one product a shopper is secretly thinking of. Instead of matching keywords against a catalog, it treats a vague request as the start of a conversation — retrieving candidates, asking one sharp clarifying question when the field is too wide, and re-ranking the instant the shopper changes their mind. It runs entirely in memory, on text alone with zero model calls on the scored path. Every decision is deterministic, offline, and instant. The intelligence lives not in a large model behind a curtain, but in the state machine, the retrieval pipeline, and the clarification logic that do the work directly.

The Problem

Keyword search punishes you for not speaking the seller's language. You type "waterproof," the listing says "water-resistant," and you get nothing back. You type "something for a beach wedding" and a filter UI has no box for that. Traditional e-commerce search treats an ambiguous query as a failure. It returns noise, or nothing, and leaves the shopper to translate their own intent into the catalog's vocabulary.

A good salesperson has none of these problems, and everything we built is an attempt to model what they do instinctively. They understand what you mean, not just what you say. They ask one sharp question when the pool is too crowded, so our agent, faced with a thousand candidates, asks the single most useful thing rather than dumping a list. They remember you as the conversation goes: what you asked for three sentences ago, that you said "nothing over $100," that you already waved away the leather ones, so our agent carries a running memory of every preference, firming the ones you repeat and quietly letting go of the ones you drop.

Our task was to build that salesperson and to prove it the hard way: surface a specific product a simulated shopper has in mind, judged on how high we rank it and how few turns we take.

Our Solution

The whole agent is built on one principle: a single structured state object that evolves dynamically. Each turn runs the same pipeline; the intelligence is in how accumulated state reshapes it. Three subsystems do the work.

The State Machine

Everything the agent knows lives in a single SessionState — slots, buy-intent, what's been asked, and history. Each slot carries a confidence score as a first-class float, and one tunable threshold splits behavior — high-confidence slots gate retrieval as hard filters, while every filled slot, hard or soft, still feeds ranking. A single number does the work an entire tier system would otherwise require.

State management is disciplined by design. Exactly one write path (bind) ever sets a value, under a total, deterministic conflict rule — a later turn overwrites an earlier one (the shopper changed their mind), and within a turn the higher-confidence reading wins. So every extractor, regex or LLM, resolves identically with no writer-specific special-casing. Crucially, the state exposes four distinct mutation verbs for four real-world meanings, not one generic clear: bind (the shopper stated it), demote (still believed, now less certain), forget (now wrong, not uncertain, used by cascades), and exclude (a negative fact kept structurally apart).

Two mechanisms make it robust. Invalidation is dependency-aware: a small graph (category → material, style, use_case, feature) drives a cascade so that changing the category transitively forgets everything scoped to the old one — cycle-safe and turn-guarded, so it clears stale slots without ever clobbering the write that just happened. And buy-intent is a continuous signal, computed from real evidence — hard-slot count, mean confidence, deflections, turn progress — with a retraction briefly capping it for one turn and self-correcting on the next. The result is a small, typed state machine with explicit named transitions and a single point of mutation — auditable module-by-module.

Dual-Track Retrieval

The single smoothed scaler intent scores drive two retrieval pipelines, selected by a smoothed buy-versus-browse signal, over three deliberately simple primitives: exact keyword search, a structured filter on an inverted index, and dense vector search — all in memory.

When the shopper is buying, category and every constraint whose confidence has crossed threshold becomes a hard filter. Studying the catalog, we found the attributes shoppers filter on hardest (color, material, size, style), mostly aren't structured fields at all; they sit buried in free-text titles and descriptions, and prices are sparse enough that a naive price filter would silently drop items. So, we run a one-time enrichment pass over all 50,000 products: parsing those attributes out of raw text into clean, normalized fields, then inverting them into an index that maps each attribute value directly to the set of products carrying it. That preprocessing is deliberately expensive, but it's where the real work goes precisely so that retrieval is nearly free. It masks the index and eliminates non-matches, collapsing 50,000 items to a small survivor set in a single lookup, with keyword and vector search ranking inside that locked subset and diversity turned down to converge on exact matches.

When the shopper is browsing, the same values demote from filters to boosts — nothing is eliminated; dense vector search leads over the full catalog, embedding the shopper's scenario rather than keywords, so a "beach wedding" query pulls dresses, sandals, and jewelry into one pool by meaning, with constraints lifting matches instead of walling off their neighbors and diversity turned up to spread across categories.

Proactive Clarification

When retrieval returns too many candidates to rank meaningfully, the agent stops and asks, but treats the choice of what to ask as a decision-tree split. Over the current candidate pool, it evaluates each unresolved attribute as a candidate split variable and computes its information gain — the reduction in entropy from partitioning the pool by that attribute's values. Just as a decision tree picks the feature that most cleanly separates its data, the agent picks the attribute whose answer would carve the pool most evenly, because that's the split that removes the most uncertainty per question. The offered options are the attribute's actual values in the current pool, so the question is always answerable and every possible answer maps to a real, non-empty branch — the agent can never ask about something the remaining candidates don't have.

The split only fires when it pays. The decision is gated hard: ask only if the pool is genuinely over-general, a positive-gain split exists, and there are turns to spare against the ten-turn budget — a clarifying question buys precision but costs a turn, so the expected narrowing must beat the turn spent. The result is that clarification behaves like growing one level of a decision tree at exactly the moments the pool is too wide to rank, and ranking directly otherwise.

What We'd Build With More Time

State management is where we'd invest first — it's the part everything else depends on. The retrieval pipeline, the clarifier, and the intent router are only ever as good as the state they read. Two upgrades were in our design from the start and lost only to debugging-and-testing time, not to doubt about their value: A Finite State Transducer for state transitions. Our current mutation logic is implemented as procedural rules. Recasting it as an FST — a legitimate NLP formalism that maps (current state, input) to (next state, output action) would make every transition an explicit, auditable, individually testable row rather than branching code. The payoff is a named contradiction state (the shopper rules out a value, then demands it) that we currently can't represent cleanly, plus a machine we can verify exhaustively instead of case-by-case. Another one, a principled intent estimator. Intent score decides which retrieval track runs, so it's one of the most consequential numbers in the system — and today it's a fresh per-turn computation. Smoothing it as a state estimate (a Kalman filter, or at minimum a confidence-weighted moving average) would stop a single ambiguous turn from flipping buying to-and-fro browsing, and the real prize give us an explicit uncertainty on the estimate, so when intent is genuinely unclear the agent can hold constraints soft or choose to clarify rather than commit to the wrong track.
Learning the thresholds instead of setting them. HARD_CONFIDENCE, the decay rate, the MMR diversity λ, and the clarify gate are all hand-tuned today. With the dev sessions as a training signal, we'd fit them to the actual scored metrics — Hit Rate, MRR, MTTC — rather than intuition, letting the data set the knobs we currently set by hand.