About the project

Shopping Copilot

A conversational shopping agent that reaches 0.990 HitRate@10 and 0.861737 TechnicalScore on the official evaluator — roughly 8× the starter baseline — with zero model calls, zero tokens, $0.00 in inference cost, no network on the scored path, and no third-party dependencies. 663 lines of standard-library Python.

I did not out-model this problem. I out-measured it. That sentence is the whole submission, and the rest of this page is the evidence.

Metric	Shopping Copilot	Official starter
HitRate@10	0.990	0.125
MRR	0.649123	0.068034
Mean turns to conversion	2.400	9.81
Efficiency	0.860	0.119
TechnicalScore	0.861737	0.106710

Official evaluator, 200 public sessions, local_evaluator.py unmodified. Reproduces byte-identically from a clean clone.

Inspiration

Nobody shops by writing a good query. People say "something for my dad's birthday" and then release their real constraints slowly — budget on turn three, battery life on turn five, and sometimes a reversal of everything they said before.

Track 4 makes that measurable in a way I found unusually honest. A simulated customer holds a hidden intent and a hidden target inside a frozen 50,000-product catalog. The agent has at most 10 turns. Each turn it either asks a question or shows ten products, and it only wins by putting the exact product in front of the customer.

$$\text{TechnicalScore} = 0.50 \cdot \text{HitRate@10} + 0.30 \cdot \text{MRR} + 0.20 \cdot \text{Efficiency}$$

What makes this hard is that the three terms fight each other. Every clarifying question buys evidence and costs a turn. Every early recommendation is cheap and usually wrong. The interesting problem is not retrieval — it's the judgment call the agent must make on every single turn: do I know enough yet, and if not, what is the one question worth spending a turn on?

That is the actual product question behind every shopping assistant I've ever used badly.

The turning point

I started where I think most people start: better semantics. Embeddings, a cross-encoder reranker, an LLM in the loop.

Before building any of it, I built diagnostics instead — read-only tools that never touch the scored path and can reject a hypothesis in minutes rather than a full evaluation cycle. One of them asks a single question: when the agent fails, was the right product ever in the candidate set at all?

The answer stopped the project cold. At the moment the agent was converting 83.5% of sessions, the target product was already inside its retrieved pool for 99.0% of them.

Retrieval was not losing the product. The agent kept finding it and then showing the wrong ten.

That reframed everything. Adding semantic retrieval would have made a system that was already finding the answer find it slightly differently. The measured bottleneck was ordering, and — the part I nearly missed — the ten-item output window itself, which was throwing away candidates the system had already earned. So I rebuilt the agent around ranking inside a pool it already had, and the score moved 0.687 → 0.818 across three tightly scoped experiments.

Embeddings and LLM reranking were never bad ideas. They were simply not where the evidence said the value was, under this task and this evaluation.

What I built

Five mechanisms, each one earning its place by measurement:

Clarification as a retrieval instrument. The agent opens with two deliberately wide, open-ended questions, then — from turn three — asks about the single catalog attribute its current top ten disagrees on most. The question is computed from the live candidate set, not read from a scripted question tree. The agent asks what would actually split its own uncertainty.

Every constraint scored on its own. A shopper's reply often carries two requirements at once. The agent splits replies at clause boundaries so each constraint scores independently against the catalog, instead of blurring into one weak signal.

Retrieve wide, rank, then cut. A 100-deep candidate pool over all 50,000 products — 70 slots scoped to the detected category, 30 unscoped as insurance against guessing the category wrong — reranked before the top ten is taken. That ordering is the whole trick: a product sitting at pool rank 80 can still reach the customer. Cut first and it never can.

Word order beats word overlap. Two products can share every one of a shopper's words and be completely different things. Ranking on the longest contiguous run of the customer's own phrasing, with plain overlap only as a tiebreak, was worth more than any re-weighting scheme I tested.

Never show the same page twice. If a turn taught the agent nothing new, its top ten is bit-identical to a page the customer already rejected. Showing it again succeeds with probability exactly zero. So the agent advances to the next window of the same ranked pool. This one is provable before it is measurable — and it is the mechanism I'd port into a real product first.

Everything is deterministic and offline. Network and external APIs were permitted in this competition; staying without them was a design choice I made because reproducibility was worth more to me than capability I couldn't audit.

How I worked

Thirteen experiments, each preregistered — hypothesis, expected channel, failure signal, and keep-or-revert rule written and committed before the code — then decided by exactly one official evaluator run. Three of them regressed and were reverted rather than kept for effort spent. Those three are documented at the same weight as the successes, because two of them are the reason later ideas worked.

Three moments taught me the most.

Two failures that were secretly one success. Splitting replies into clauses scored −0.004 on its own. Front-loading open-ended questions scored −0.002 on its own. Under a strict one-change-at-a-time discipline I would have reverted the first and never run the second. Together they scored +0.021. Acquiring constraints faster is worthless if your scorer collapses them into one signal; finer resolution is worthless if constraints arrive slowly. They were complements, not additions. Bundled deliberately as one indivisible experiment, they cut mean turns-to-conversion by nearly a full turn — sessions solved by turn two went from 38 to 94, and the long tail past turn six disappeared entirely.

A change I could prove before I ran it. Advancing the page on an idle turn cannot lower HitRate@10 and cannot raise turns-to-conversion, because the repeated page has already been judged. The measurement matched the argument exactly: six sessions recovered, and not one session lost across all 200. Four of those six were precisely the four sessions my previous experiment had broken and left unexplained — the fix and the wound were separated by one experiment and I only connected them by reading per-session transitions rather than aggregates.

Knowing when a direction is finished. The same diagnostic that opened the ranking direction later closed the retrieval one: at pool depth 50 the agent was converting 99.5% of everything a perfect reranker could convert there. Recall headroom was worth 0.0025 of the final score. Knowing a direction is exhausted is as valuable as knowing one is open, and much rarer to have evidence for.

Where it falls short

Evidence is append-only — the agent never truly retires a superseded preference. I tested explicit erasure on intent-override and it made things measurably worse, so I reverted it rather than shipping tidiness. The honest middle ground, supersession at clause granularity, is untested and is the first thing I'd build next.

Two of my mechanisms lean on the simulator's own reply semantics. That is a real coupling, disclosed in my limitations rather than dressed up as pure shopper insight — understanding the interaction protocol turned out to be part of the engineering problem, but I don't get to pretend it's all product intuition.

And these are 200 public sessions with no confidence interval. A gain concentrated in six sessions is a different class of evidence than a dominance proof, and I labeled every result with which class it belongs to.

What I learned

The most valuable thing I built isn't in the submission. It's the diagnostic layer — the tooling that answers why did this fail before I spend a cycle guessing. Two reverted experiments each cost a full build-evaluate loop to learn a single number. The diagnostics gave me the same answers in minutes, and every good decision after that traces back to them.

Diagnoses expire. "Ranking, not recall" was true at one pool depth and meaningfully wrong at another. A bottleneck is a property of a configuration, not of a problem — and re-running the diagnosis after every structural change caught me being wrong twice.

Composite metrics have to be decomposed before you optimize them. Some of my changes could only move one term, by construction — one couldn't touch HitRate@10 at all; another paid back a fifth of an earlier gain to buy coverage. Knowing which term a change can move, before running it, converted a series of surprises into a series of confirmations.

Sophistication is a cost, not a virtue. I'd have enjoyed shipping an LLM-in-the-loop reranker more. But the version I shipped runs 200 full conversations in seven minutes on a laptop for nothing, reproduces bit-for-bit, and let me run thirteen honest experiments instead of five noisy ones. On this problem, the discipline was the differentiator — and I suspect that's true of considerably more than this problem.