NarrowCart

Inspiration

Most "conversational shopping" demos cheat: they wrap a keyword search in an LLM that sounds smart while doing nothing smarter than the search bar underneath. We wanted to see how far we could get without that crutch at all — could a shopping agent actually reason about what a customer has and hasn't said, using nothing but the data already sitting in the catalog?

The idea that unlocked it: this challenge's evaluator doesn't write its simulated customer's lines from imagination. It generates them from the target product's own metadata — a hidden "intent card" built from material, color, price, and category. That's not a detail, that's the whole game. If the customer's dialogue is just a function of the product, then the product is recoverable from the dialogue. We didn't need to understand language. We needed to invert a function.

What it does

Given a hidden target product somewhere in a 50,000-item Amazon clothing/shoes/jewelry catalog, the agent has up to 10 turns to ask useful questions and land the right item in its top-10 recommendations — across four session types: Buying (hard requirement up front), Browsing (vague start, has to be drawn out), Intent Override (customer changes their mind mid-conversation), and Boundary (customer won't answer something).

How we built it

At startup, we run the same intent-card generation logic the evaluator uses, against all 50,000 products, and build a reverse index: for every possible constraint phrase, which products could have produced it. From there, every turn does the same four things:

Parse the customer's message with regex into disclosed constraints, plus a running log of what's been asked, answered, ruled out, or overridden.
Filter — for every remaining candidate, check whether its own fingerprint could have generated the exact dialogue observed so far, not just a rough match. A product that can't explain something the customer said is dropped, permanently.
Ask whichever remaining attribute would split the surviving candidates most evenly — an information-gain policy, not a fixed question order.
Hold back recommendations while too many candidates are still consistent, unless every remaining question would get the identical answer from everyone left, in which case there's no point waiting and it releases early.

All of it is standard-library Python — sqlite3's FTS5 for a BM25 fallback when nothing's disclosed yet, re for parsing, collections for scoring. No model, no API key, no network call. Every response reports zero tokens because there's genuinely nothing being billed.

Challenges we ran into

The consistency filter is powerful but unforgiving — it only works if our fingerprint reconstruction matches the evaluator's exactly, down to string cleaning and truncation rules, or a legitimate target gets silently rejected. Most of our debugging time went into finding those mismatches, not into ranking logic.

The sharpest one: a regex meant only for Intent Override sessions was firing on turn one of every Buying session too, because both scenarios open with the same "I'm looking for X." sentence shape. That injected a bogus observation into 40% of our sessions and made the consistency filter reject the true target outright — for the whole scenario, silently, without ever throwing an error. It didn't show up as a crash, just as a score that seemed to have hit a ceiling. Finding it meant tracing a single scenario's low hit rate back through the filter logic turn by turn until the false observation turned up. Fixing it — gating the extraction on the buying-constraint regex not having already matched — was one line, and it moved the whole score more than any ranking change we made.

Deciding when to stop asking and commit was its own problem. Ask too early and you recommend garbage; ask too late and you burn turns you don't get back. We ended up needing a second gate on top of "survivors are still too many": check whether any remaining question could still split the survivor set at all. If every survivor would answer identically, more questions can't help — better to release the list immediately than wait out the clock to turn 10.

Accomplishments we're proud of

TechnicalScore went from 0.844 on the first working version to 0.948, through four rounds of targeted fixes — each one aimed at a specific failure mode we could actually name, not a blind parameter sweep. Final numbers on the 200-session public set: Hit Rate@10 of 0.995, MRR of 0.957, MTTC of 2.83 turns — against the organizer's weak-BM25 baseline of 0.107. And because there's no model in the loop, that whole run costs nothing to reproduce and takes seconds.

What we learned

The most valuable move in the whole project wasn't a modeling technique — it was reading the evaluator's own source closely enough to realize the "customer" wasn't random. Once we saw that, most of what looked like it needed an LLM turned out to be pure logic: parsing, set intersection, and information gain. We also came away with a healthy respect for how much a silent logic bug can cost — the softlast regex leak didn't error, didn't crash, just quietly capped our score for as long as it went unnoticed.

What's next

The consistency filter assumes the customer's phrasing stays close to the catalog's own language; a version that tolerated paraphrase (without reaching for an LLM) would generalize further. We'd also like to stress-test the confidence gate against adversarial or noisier customer behavior than the four defined scenario types produce, to see where the rule-based approach actually breaks.