The Powerpuff Girls

Inspiration

The challenge starts you off with a weak BM25 baseline that only finds the hidden target product 12.5% of the time. That gap between "technically works" and "actually useful" is what pulled us in. A shopping agent that can't ask a good follow-up question is basically just a search bar with extra steps and we wanted to see how far session state, a real clarification strategy and careful re-ranking could push that number without reaching for an LLM at all.

What it does

A multi-turn shopping agent that holds a conversation with a customer. It asks natural clarifying questions about material, colour, budget, style and remembers everything disclosed across turns and returns a ranked top-10 product guess on every single turn so that it never goes silent while waiting on an answer. It runs entirely deterministically with zero API calls and zero marginal cost per session.

How we built it

Retrieval sits on a SQLite FTS5 index over all 50,000 catalog products, using a field-weighted BM25 query, with titles weighted highest, then categories, features and details, store and description. On top of that we track three separate term sets per session, category terms fixed on turn one that survive intent overrides, learned terms and genuinely disclosed constraint terms, rather than lumping everything into one flat bag. That separation is what makes it safe to handle a customer changing their mind mid-conversation. We then fuse BM25's ranking with a second term-coverage score using Reciprocal Rank Fusion, so candidates matching more of what the customer actually said get pulled up without ever losing recall. Clarifying questions walk a fixed, empirically tuned attribute order and stop once turns run short or answers stop being informative. Everything runs on the Python standard library, so there's no numpy, no vector database and nothing that needs a network connection at inference time.

Challenges we ran into

The hardest part wasn't coming up with ideas, it was killing the bad ones. Several changes that looked like obvious wins on paper regressed once we actually measured them, things like filtering out boilerplate reply text, stopping clarification once the candidate pool stalled, or picking questions by information gain instead of a fixed order. The reasons only showed up once we dug into the failure cases. For example, filler words like "additional" turned out to carry real signal because they also show up in ordinary Amazon listing copy. We also ran into a hard ranking ceiling on near-duplicate products, like a leather belt buried at rank 226 among 258 near-identical belts which pure keyword scoring has no way to break without a semantic signal we deliberately chose not to build in.

Accomplishments that we're proud of

Taking HitRate@10 from 0.125 to 0.905 and TechnicalScore from 0.107 to 0.771, roughly a 7x improvement, without a single API call. Every session runs at zero cost and behaves identically whether or not there's a network connection which directly answers the spec's warning that network access may be disabled for final scoring. Sub-100ms p99 latency also meant we could re-run the full 200-session evaluation in under a minute after every change, so nearly every decision in this project is backed by a measured before and after rather than a guess.

What we learned

That a negative result, properly measured and root-caused, is worth writing down alongside the wins. We reverted 5 of 7 experiments this cycle and each one taught us something about the evaluator's actual behaviour that the next idea built on. We also learned how much leverage there is in modelling conversation state carefully, tracking what's been asked, what's been disclosed, and what survives an intent override, before ever reaching for a bigger model.

What's next for The Powerpuff Girls

Adding a lightweight semantic or embedding signal to break the near-duplicate ranking ceiling, learning the clarifying-question policy from real customer responses instead of a fixed hand-tuned order and validating our override and no-preference language detection against a much wider range of phrasing before trusting it on the private evaluation set.