Inspiration
Every "AI shopping assistant" demo we had seen worked the same way: paste the whole conversation into an LLM, hope the right product comes back, pay per token. When we read the Track 4 kit, two things stood out. The organizer's own baseline which is a plain BM25 retriever, scored 0.1067, and the submission rules warned that final scoring may run with network access disabled.
That reframed the problem. The winning move was not a bigger model; it was an agent that could reason about a 50,000-product catalog on a CPU, offline, for free. So we set ourselves a constraint on day one and never relaxed it: no network, no tokens, no API bill and beat the LLM-shaped alternatives anyway.
What it does
ByteMe Shopping Copilot holds a multi-turn conversation with a shopper, narrows down what they actually want, and returns ten ranked products every single turn.
On the 200 public evaluation sessions, measured with the official local evaluator:
| Metric | Organizer baseline | ByteMe |
|---|---|---|
| Hit Rate@10 | 0.125 | 0.985 |
| MRR | 0.068 | 0.824 |
| Mean turns to conversion | 9.81 | 2.41 |
| TechnicalScore | 0.1067 | 0.9114 |
That is 8.5× the baseline, at 0 tokens and $0.00 per session, with the network switched off. Index build takes ~19 seconds; a full session runs in ~140 ms. It holds up across all four scenario types — buying, browsing, intent override, and boundary sessions where the shopper genuinely has no preference.
How we built it
The agent is four stages, all in Python's standard library.
1. Dialog state, not a text blob. The shipped baseline re-queried from scratch on every turn, throwing away everything the shopper had said. Ours accumulates constraints into slots, and — critically — when a shopper walks something back, it retracts only that preference, not the whole slot set.
2. Multi-route candidate generation. Three retrievers run in parallel and are fused:
- a category lattice that resolves the shelf the shopper named, indexing 1- and 2-component windows plus intersections of individual components, so non-adjacent paths like "Women … Bodysuits" still resolve;
- a phrase index of listings that literally make the claim the shopper stated;
- field-weighted BM25 over the whole dialog as a lexical backstop.
3. Rarity-weighted reranking. Claim matches are weighted by $\log\left(1 + \frac{N}{df}\right)$, so "cotton" on a 1,200-product apparel shelf counts for almost nothing while "brushed nickel" counts for a lot. Without this, common materials flooded broad shelves and buried the target.
4. An information-gain clarification policy. The agent opens with a broad question, then picks the facet that maximally splits the remaining candidate set — and stops asking once questions stop paying. Recommendations and the question ship in the same response, so all ten slots are always filled.
We also built a two-tier message parser. Tier 1 matches the organizer's template phrasings. Tier 2 runs only on what tier 1 missed, keyed on structure rather than vocabulary: a payload is what follows a colon; a refusal is a negation with no payload, whose target attribute comes from the question we just asked rather than the shopper's wording.
Around the agent: a scenario-sliced scoreboard, a failure-analysis harness that separates reachable / surfaced / ranked, a weight sweeper, a paraphrase robustness harness, and 39 unit tests.
Challenges we ran into
Unreachable shelves. Four categories in the public set could not be resolved at all by naive category matching, the shopper's phrasing skipped levels in the taxonomy. The lattice union fixed it and took unreachable shelves from 4 to 0.
Intent overrides. When a shopper changes their mind mid-session, our first implementation dropped every stated preference. Retracting only the contradicted one took intent-override Hit@10 from 0.433 to 0.933.
The public set cannot measure the thing we most wanted to measure. Our robust parser and the template parser differ in 0 of 200 sessions because the public sessions use literal template scaffolds, the paraphrase-tolerant path is never exercised. On a paraphrase harness we wrote ourselves, template-only parsing retained just 42% of clean score; the structural tier brought that to 89%. We shipped it knowing the public scoreboard would show a tie, because the hidden set is 800 different sessions.
An exploit we chose not to take. We found a degenerate configuration worth +0.0042 that games how the evaluator scores unconstrained turns. We measured it, wrote it up as an ablation row, and did not ship it. It would not work on a real shopper.
Accomplishments that we're proud of
- 0.9114 TechnicalScore with zero tokens and zero network calls. The cheapest architecture and the most deployable one turned out to be the same architecture.
- Every claim in our README is a measured number, re-verified from a clean clone: fresh clone → catalog download → official evaluator, no manual fixes.
- We report our negative results. A constraint-coverage bonus that cost 0.08 turns for no gain. Override demotion, shipped, that is 0.0006 behind plain accumulation. The exploit we declined. All of them are in the ablation table.
- Boundary sessions at 1.000. The hardest slice to get right is knowing when not to ask another question.
What we learned
Reading the evaluator line by line was worth more than any modeling decision — it told us the shopper was a deterministic function, which made our whole iteration loop seconds instead of hours.
Held-out or it didn't happen. We k-folded the public set and vetoed any change that only helped in-sample; three "improvements" died that way.
And the discipline that mattered most: don't tune to the simulator. We know exactly which parts of our score come from real retrieval quality and which parts the simulator hands us for free, and we say so in the README rather than hoping nobody checks.
What's next for ByteMe Shopping Copilot
- An optional LLM layer, behind a flag, with the offline ranking as a deterministic fallback — a Gemini-backed parser and session memory are already prototyped. The offline path stays the default and stays the number we report.
- A learned reranker over the fused top-100. All 9 remaining misses are ranking failures, not recall failures — the target is on the right shelf and matches every stated claim, just outranked. That is a supervised problem, and it is where the last ~0.02 lives.
- Cross-session memory, so a returning shopper's known constraints seed turn one.
- A real catalog and real shoppers. Everything above is architecture that survives contact with actual phrasing which is exactly what the structural parser tier was built for, and exactly what a simulated shopper cannot prove.
Log in or sign up for Devpost to join the conversation.