Hunch shopping agent

A shopping agent that shows one product instead of ten when it is unsure. Hunch's confidence is a measured quantity, AUC 0.845. 50,000 items to the right one in 2.29 turns, offline, zero dependencies.

Comment

Inspiration

Every product search I have used does the same thing when it does not understand me: it returns ten results anyway. The list is never empty, never uncertain, never says I think it's this one, check me? It is confident by construction, because ten results is what the page expects.

Track 4 makes that habit measurable. A simulated shopper opens vaguely, "I'm looking for pants", and the agent has ten turns to surface one product out of fifty thousand. Then the scoring formula does something unusual:

$$\text{Score} \;=\; 0.50\,\text{HR@10} \;+\; 0.30\,\text{MRR} \;+\; 0.20\,\text{Efficiency}$$

Reciprocal rank is scored once, at the turn the target first appears, and the session stops there. So putting products on screen is an irreversible spend. A hit at position seven is not "a hit", it is permanently $\tfrac{1}{7}$.

I priced the two moves against each other. One extra turn costs

$$0.20 \times \frac{1/10}{200} \;=\; 0.0001$$

while lifting one session from rank 3 to rank 1 gains

$$0.30 \times \frac{2/3}{200} \;=\; 0.0010$$

The exchange rate is 10:1 in favour of spending turns to buy rank. If a turn is cheap and a bad rank is permanent, the right move when you are unsure is to say less.

What it does

Hunch computes margin, how far its leading candidate is clear of the runner-up. That one number separates "top-1 is the target" from "top-1 is not" at AUC 0.845. Two intuitive alternatives came out at 0.533 and 0.476, coin flips. Only margin is real, and the agent is named after it.

Strong margin, it fills all ten slots. Weak margin, it shows one product and asks a question. Here is a real turn from session public_0004, where the margin collapses to 0.00 and the agent says so out loud:

"I am searching for a long torso polyester camisole, but this single best guess is actually socks, and I would rather check than pad the list. Do you have any other requirements?"

It recovers on the next turn and lands the target at rank 1.

TechnicalScore  0.963303      HR@10  1.000      MRR  0.963345
MTTC  2.29 turns              misses 0 / 200    ~25 ms per session
zero third-party dependencies                   runs offline with no API key

How I built it

Everything is a switch. One Config dataclass holds every weight and every ablation flag, and the ablation runner prices each component by handing the pipeline a modified config. A component that cannot be switched off cannot be priced, and cannot honestly go in a report. Thirty-six arms, each with a paired bootstrap 95% interval against a measured noise floor of $\pm 0.0268$.

Score What changed
Official weak baseline 0.1067 stateless BM25, never asks
+ one line: ask "other" 0.4970 the shopper starts answering
+ three lines: keep the history 0.7504 my real baseline
+ retrieval and state machine 0.8838 four channels, decay over erasure
+ confidence-gated display 0.9506 one product when unsure
+ catalog grounding 0.9505 free here, buys robustness
+ one re-weighted prior 0.9633 a parameter nobody had swept

I report against 0.7504, not 0.1067. Against that patched starter the paired gain runs +0.1873 to +0.3172 across five customer wordings, all five intervals excluding zero, widest on the chattiest shopper where keyword matching degrades first.

The four pillars, including the one I disagree with:

Pillar What I did
Multi-Route Retrieval Four channels fused by weighted sum rather than intersection: aisle hard filter (50,000 to median 184), idf-weighted exact-phrase index over 225,670 real feature strings, BM25, purchase-likelihood prior. Intersecting on every accumulated requirement empties the candidate set on 139 of 200 sessions
State Machine + Intent Override Requirements accumulate with belief weights; a retraction decays them to 0.5 rather than clearing. Erasing costs −0.0514. I also built the targeted replacement the webinar calls strong-agent behaviour, then checked whether the contradiction it repairs exists here. It does not: the "withdrawn" preference is still literally true of the target in 28 of 30 sessions, so replacement measures as a tie
Runtime Adaptation The confidence gate. The system adapts how much it says to how well separated its own evidence is
Dual-Track Routing Built, measured, reported as falsified. The explorer-vs-buyer classifier is real and stable across all five wordings, but wired into ranking it is worth +0.0000. For ranking, both customer types want the same thing. The switch stays in so the negative result stays reproducible

Challenges I ran into

My "understanding" layer turned out to be a phrasebook. Rewording the customer without changing a single requirement took the score from 0.847 to 0.765. Not degradation, collapse: category parsing fell 200/200 to 0/200, extracted requirements 1.63 per session to 0.00. There was no understanding in that layer, only templates matched against one speaker's sentences. I rebuilt it against the catalog's own strings, which is measurably negative on the public wording and cut cross-wording spread from 0.1435 to 0.0053.

I spent a day testing the wrong thing. Five paraphrase arms, all holding comfortably. Then I built an instrument that splits a paraphrase into the carrier sentence and the requirement itself, and found all five arms had only ever rewritten the carrier. The real exposure had never once been under test. That instrument went on to reverse my verdict on the reranker.

Deciding what not to ship was harder than building any of it. The cross-encoder went around three times: ruled out at −0.1120 on a crude first measurement, switched back on at +0.0343 when a better one arrived, then off again when a change to an unrelated parameter turned it significantly negative, without a line of it changing.

Accomplishments that I'm proud of

Zero misses. HitRate@10 is 1.000 across all 200 public sessions, in every one of five customer wordings, with a mean of 2.29 turns to the first hit.

Zero third-party dependencies. requirements.txt is empty by design, including on the LLM path, where the client is urllib.request from the standard library. An openai or httpx that will not install in a restricted scoring environment turns a working entry into a zero. Verified with the organizer's own CLI from a clean directory, on two interpreters and an empty virtual environment, bit for bit.

Every component carries a price, including the ones that lost. Three ideas were falsified and ship switched off with their numbers attached. One default was reversed only after I stopped asserting its failure path and measured it.

What I learned

1. This benchmark does not want a semantic layer, and that is provable. 94.2% of everything the shopper says appears character for character inside the target product's own text. When someone says "100% Polyester", that exact string is printed in the target's feature list. There is nothing to disambiguate. I built three semantic layers anyway and priced all three where lexical matching genuinely fails:

Layer Effect Verdict
Cross-encoder reranker −0.0119 significant loss
Dense vector retrieval +0.0024 indistinguishable from zero
LLM constraint translator −0.0003 indistinguishable from zero

2. A component-ablation table cannot see a parameter that is never switched off. My largest single improvement was one float. w_prior was set to 0.9 early and never swept, while the rest of the system was built around it. Every ablation arm switches a component off; this one was never off. Swept on the training half, validated once on a 60-session holdout: +0.0128 overall, +0.0117 on the holdout, a train/holdout gap of 0.0015, well inside the noise floor.

What's next for Hunch shopping agent

The largest known hole is the synonym exposure, −0.0404, where the shopper words a requirement the catalog does not use. An offline lexical expansion grounded in the catalog's own co-occurrence structure has not been tried, and it would close the gap without a model dependency.

Second, category parsing assumes a shopper who names a product type. On three hand-written human openers it parsed 0/3, leaving the pool at 50,000 instead of ~184. The option-bearing question I added for that case fires 0/200 times on the official set and 3/3 on the human ones. That exposure is real for real users and invisible to this benchmark, which makes it the one I would fix first for production.

Third, every dialogue-policy conclusion here rests on one simulator's reply policy. ask_attribute="other" is optimal against this responder. value.py re-derives it from the catalog rather than from the simulator, which is evidence it is not purely an artefact, but a second responder model is the only way to know.

Built With

Share this project:

Updates

Submission history