Power Rangers

Inspiration

Online shopping search works well when users know exactly what they want, but real conversations are rarely that simple. People change their minds, add new requirements, remove old ones, and describe products in different ways.

We wanted to build a shopping copilot that could actually follow this conversation instead of treating every message as a completely new search.

What it does

Our project is a conversational product recommendation system that keeps track of a shopper's preferences across multiple turns.

It understands constraints such as product type, colour, style, use case and other requirements, while also handling situations where the user changes their mind. The system then searches through a catalogue of 50,000 products and ranks the most relevant results.

Instead of relying entirely on an LLM to generate recommendations, we combine structured intent tracking, deterministic retrieval and ranking with LLM-based language understanding. This helps us keep the system accurate while still being able to understand natural and varied user input.

How we built it

We started with the provided BM25 retrieval baseline and studied where it failed, especially when conversations contained multiple constraints or changing user intent.

From there, we built a stateful search pipeline that:

Extracts and tracks user requirements across turns
Detects when requirements are added, removed or replaced
Uses weighted retrieval and ranking to find relevant products
Handles exclusions and negative preferences
Uses an LLM to better understand paraphrased and less structured language
Returns catalogue-valid product IDs ranked by relevance

We repeatedly evaluated the system using the official evaluator and also created additional synthetic and paraphrased test cases to test behaviour outside the public examples.

On the official 200-session evaluation set, our final system achieved a 100% Hit Rate@10, 0.953 MRR and a 0.965 technical score, around 9x the baseline technical score.

We also used split-half validation when tuning important ranking weights to check that improvements were not simply caused by memorising the public evaluation set.

Challenges we faced

One of the biggest challenges was balancing flexibility with reliability.

An LLM is useful for understanding natural language, but allowing it to control the entire recommendation process can make results inconsistent. On the other hand, a purely rule-based system struggles when users phrase the same requirement in completely different ways.

We therefore had to decide carefully which parts should be handled deterministically and where an LLM actually added value.

Another challenge was conversational state. A user might first ask for a black jacket, then say "actually, make it blue", or later remove a previous requirement. Correctly deciding which preferences should remain active turned out to be just as important as retrieval itself.

What we learned

The biggest lesson was that strong conversational search is not just about having a better retrieval algorithm. It is also about maintaining an accurate understanding of what the user currently wants.

We also learned the importance of testing beyond the provided examples. A system can perform extremely well on a public dataset while still being fragile to paraphrasing or slightly different conversations.

Building additional stress tests and validating our changes helped us focus on improvements that were more likely to generalise rather than simply increasing a single benchmark score.