title

Inspiration

We began with a practical question: how effective can a shopping agent be without an LLM, a GPU, or paid API access?

The challenge required the agent to identify a hidden product from a catalog of 50,000 items through a conversation of up to ten turns. At the beginning, we had limited experience with agent development, information retrieval, and conversational systems. We therefore started by studying the evaluation protocol and the local evaluator. This helped us understand how user messages were generated, how recommendations were scored, and why Hit Rate@10, MRR, and MTTC mattered.

This analysis led to an important conclusion: retrieval quality alone would not be enough. The agent also needed to preserve useful information across turns, respond to changing preferences, and decide when asking another question was more useful than returning more products.

What We Learned

Dialogue State Matters

Our first major lesson was that a conversational shopping agent should not treat every user message as an independent query. It needs to maintain a session-level representation of the user's evolving intent.

The agent therefore accumulates meaningful terms across turns and updates its candidate set as new information becomes available. This allows an initially vague request to become more precise without discarding useful context from earlier turns.

We also tested several intuitive strategies that reduced performance. Clearing all previous terms after an intent override removed useful category information. Adding every profile preference to the query introduced generic and noisy terms. These experiments showed that state management must be selective: the system should remove information only when it is clearly no longer useful.

Real Shopping Is an Iterative Process

In a real shopping interaction, users rarely provide all their requirements in the first message. They may begin by browsing broadly, notice differences between products, and then decide that material, colour, features, or intended use matter more than they initially expected. They may also change their mind after comparing several options.

For this reason, our dynamic questioning strategy was designed to model a realistic shopping process, not only to satisfy the required number of conversation turns. Instead of following a fixed question order, the agent reassesses the current candidates after each turn and asks about the attribute that is most likely to help the user make a decision.

How We Built It

Dual-Layer Dynamic Architecture

The system has two interacting layers.

The outer layer routes the session between two retrieval modes. When the accumulated request contains relatively little information, the agent uses a browse mode that combines keyword retrieval, synonym expansion, category matching, profile signals, and popularity. This provides broader recall for exploratory requests.

When the request becomes more specific, the agent switches to focus mode. This mode relies mainly on BM25 keyword retrieval and category matching to produce a more precise candidate set. Results from the active retrieval signals are combined using Reciprocal Rank Fusion and then reranked using term coverage, Jaccard similarity, and an unseen-item preference.

The inner layer controls clarification. After every retrieval step, the agent examines the current candidate pool and estimates the value of asking about each available attribute:

$$

\text{QuestionValue(attribute)}

\text{NormalizedEntropy(attribute)} \times \text{Coverage(attribute)} $$

Entropy measures how differently an attribute is distributed across the candidates. Coverage measures how many candidates contain a recognizable value for that attribute. An attribute is therefore considered useful when it both distinguishes the candidates and appears often enough to support a meaningful question.

The controller can select attributes such as material, colour, style, use case, or feature. It repeats this evaluation on each turn and stops asking questions when the candidate pool is sufficiently small, when the turn limit is reached, or when no remaining attribute provides enough information.

Conversation State and Preference Changes

For each session, the agent stores:

accumulated query terms;
previously asked attributes;
the most recently asked attribute;
already displayed product IDs; and
profile terms available at the start of the session.

When the user changes their mind, the agent keeps context that may still be useful, clears the displayed-item history, and reopens the most recently asked attribute. This allows the new preference to influence the ranking without losing the product category or other relevant information.

When the user says that they have no preference for an attribute, the agent avoids repeatedly asking the same question and moves on to another possible source of information.

Challenges

Silent Failures in the Evaluator

One difficulty was that the evaluator catches exceptions and converts them into empty responses. During development, a single undefined variable caused the agent to return no recommendations without producing an obvious error in the final score.

To address this, we tested individual calls before running the full evaluator, recorded complete conversation traces, and examined results separately for each scenario type. This was especially useful for diagnosing intent overrides and boundary responses.

Semantic Correctness Does Not Always Improve Retrieval

An intent override seems to suggest that all previous preferences should be removed. However, our experiments showed that some earlier terms, particularly category terms, remained useful after the change.

The final implementation therefore retains useful context while resetting the displayed-item state and reopening the relevant question. This is closer to how a user would normally revise a shopping request: a changed colour preference does not necessarily mean that the user has changed the product category.

Ranking Is Harder Than Recall

The agent usually finds the correct product within the top ten, but the correct item is not always ranked first. This creates a gap between Hit Rate@10 and MRR.

Improving this ranking remains a challenge because many catalog items share similar keywords. Future work could explore a softer penalty for previously shown items, more detailed attribute matching, or a lightweight embedding-based reranker.

Technology Choices

The implementation uses Python 3.10+ and the standard library. Its main components are:

SQLite FTS5 for the in-memory full-text index;
BM25 for keyword ranking;
Reciprocal Rank Fusion for combining retrieval signals;
a hand-written synonym map for lightweight query expansion;
Jaccard similarity for term-based semantic comparison;
Shannon entropy and attribute coverage for question selection; and
regular expressions and session state for intent and dialogue management.

We deliberately did not use an LLM or an embedding model. Loading an embedding model and encoding every query would add model-loading overhead, per-turn latency, memory usage, and deployment dependencies. For a ten-turn interactive task, we preferred a fully offline system with predictable response times, zero token usage, and no third-party packages.

Results

The current version was evaluated on 200 public sessions. The technical score is calculated as follows:

$$

\text{TechnicalScore}

0.50 \times \text{HitRate@10} + 0.30 \times \text{MRR} + 0.20 \times \text{Efficiency} $$

The final results were:

Hit Rate@10: 0.990
MRR: 0.700702
MTTC: 3.005
Technical Score: 0.865111
Token usage: 0
Model cost: $0

The system runs entirely offline and does not depend on an external API, GPU, or third-party Python package.

Future Improvements

The most direct improvement would be more precise handling of intent overrides. During development, we explored attribute-aware decay, where a superseded value would be down-weighted instead of treated as completely irrelevant. This approach was not included in the final submission, but it could improve ranking in override sessions.

Another possible direction is a lightweight embedding-based reranker. It could improve paraphrase handling and ranking among products with similar keyword matches, provided that the additional memory and latency remain acceptable for offline deployment.

This project taught us that a useful shopping agent requires more than a retrieval model. It must track the conversation, recognize that user preferences can change, decide when clarification is worthwhile, and balance exploration with precision. The resulting system is intentionally lightweight, but its interaction strategy is designed around the way people actually shop: they explore, compare, refine their requirements, and sometimes change their minds.