Agent Dory - Conversational E-Commerce Search

• 

  UI Demo for 4 Sample Test Cases

Inspiration

Like Dory searching the ocean for Nemo, Agent Dory searches a sea of products for the right one. Customers rarely know the perfect search query upfront, so conversational search lets them provide clues gradually, clarify preferences, and change direction while the system keeps searching for Nemo—the hidden target product.

What it does

Agent Dory is a conversational e-commerce search agent for a frozen catalogue of 50,000 products. It remembers the conversation, extracts useful product clues, asks follow-up questions, and continuously updates its recommendations.

The local evaluator simulates the customer and measures whether Dory can find Nemo within ten turns.

How we built it

We built the system as a local, reproducible retrieval pipeline:

• A rule-based parser tracks conversation state, categories, facets, preferences, and intent overrides.
• SQLite FTS5/BM25 provides broad lexical retrieval using an OR query, so one incomplete term does not eliminate every result.
• Hugging Face's naver/splade-v3-lexical model provides learned sparse lexical ranking. It expands and weights vocabulary terms while remaining grounded in product text.
• Catalogue embeddings are generated once, cached locally, and reused. SPLADE ranks only rows that pass the eligible category and facet filters.
• If SPLADE is unavailable, the system falls back to FTS5/BM25 and should still provide a strong result.
• We implemented two question strategies: always_other and Generalized Binary Search. We primarily use always_other because the evaluator’s customer has useful answers for only a small number of attributes. Asking a specific question can produce a “no additional preference” response even when another useful clue remains undisclosed. Asking "other" gives the evaluator a chance to reveal any remaining constraint, making each turn more likely to improve retrieval. We keep GBS as an alternative because it is mathematically useful for selecting informative questions, even though some of those questions may not receive useful answers from the scripted customer.
• Recommendations rotate across turns so the system does not repeatedly show the same products. After an intent override, previously shown products are added back because the customer's intent has changed.
• We avoided an external LLM so the system remains inspectable, reproducible, and usable without an API key.

Challenges we ran into

The hardest problem was figuring out why the agent kept missing the target. The evaluator has special scripted behavior: it reveals clues according to a hidden intent, sometimes gives no useful answer, may provide incomplete wording, and does not tell the agent which hidden facts are considered important.

We had to balance strict and flexible retrieval. FTS5 AND was too strict because one truncated or unusual term could remove all results. OR preserved recall but introduced noise. We also had to understand how SPLADE, hard filters, conversation state, recommendation rotation, and intent overrides interacted.

At first, a benchmark score alone did not explain the misses. To solve that, we created an additional benchmark and miss-inspection workflow that replays failed sessions one by one and records the query, filters, retrieval source, candidate ranks, SPLADE status, and recommendation history.

Accomplishments that we're proud of

We learned how to use a pretrained Hugging Face model in a real retrieval pipeline rather than treating it as a black box. We integrated SPLADE-v3 through Sentence Transformers, built catalogue-vector caching, filtered eligible products before scoring, and preserved BM25 as a strong fallback.

Our research into information retrieval helped us understand why SPLADE is a useful middle ground: it combines learned lexical expansion with sparse, vocabulary-based representations. The original SPLADE work emphasises retaining the useful properties of lexical retrieval, while SPLADE-v3 reports strong comparisons against BM25 and other retrieval systems.

• SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking
• SPLADE-v3: New baselines for SPLADE

Our main research focus was Generalised Binary Search and information-gain question selection. We studied how GBS chooses queries that split the remaining hypotheses as evenly as possible, and how entropy can be used as a greedy approximation to minimising the number of questions. That directly influenced our GBS policy, which scores candidate attributes from the current product pool and asks about the most informative one.

• The Geometry of Generalized Binary Search
• Query Learning with Exponential Query Costs
• Optimal sets of questions for Twenty Questions

We also reviewed conversational search research as supporting background, especially work on context-dependent queries, omission and coreference, query reformulation, dense retrieval, and mixed-initiative questioning. However, our system differs from a typical conversational search setting: the evaluator's customer responses are scripted from a hidden intent rather than being fully dynamic or open-ended.

In a real shopping conversation, it may be natural to ask, “Do you want a shirt or jeans?” Here, the goal is more mathematical: choose the question that most effectively splits the remaining product candidates. This is why Generalized Binary Search is useful for our system. It selects questions based on information gain and candidate separation, even when the question may feel less natural in a normal conversation.

These conversational-search ideas still influenced our decision to maintain structured conversation state and use follow-up questions instead of relying on a large language model to rewrite every turn.

• Explicit Query Rewriting for Conversational Dense Retrieval
• Mixed-initiative Query Rewriting in Conversational Passage Retrieval

Most importantly, we turned unexplained benchmark failures into observable engineering problems. The miss reports made it possible to inspect what the agent believed, what products were eligible, which retriever was used, and where the target disappeared.

What we learned

We learned that conversational search is not just ordinary search with more messages. The difficult part is deciding which parts of the conversation should remain active, which clues should be added to the query, and how to handle a customer who says “actually, I changed my mind.”

We also learned that retrieval quality depends heavily on query construction. A powerful model cannot fully compensate for an incorrect or overly strict query. Sparse lexical retrieval is valuable because it offers more flexibility than exact matching while remaining more grounded and inspectable than a fully dense semantic search system.

Finally, we learned that benchmark-specific behavior matters. A theoretically optimal question is not always the best question if the evaluator cannot answer it. In this setting, asking other can reveal more useful information than asking a mathematically optimal facet question.

What's next for Agent Dory - Conversational E-Commerce Search

Because the evaluator uses fixed and highly structured customer phrasing, we believe the current rule-based parser and query construction are already close to optimal for this benchmark. Training a larger model specifically on these phrases would likely provide limited benefit and could simply overfit to the evaluator.

For real customer queries, the language would be much more varied, indirect, incomplete, and unpredictable. In that setting, an LLM or a learned conversational query-reformulation model could help resolve intent and convert natural language into better search terms. That broader setting would also show SPLADE's effectiveness more clearly, because its lexical expansion and ranking strengths would be tested against genuine vocabulary mismatch rather than mostly fixed evaluator phrasing.

The next step would be to evaluate BM25 and SPLADE on more realistic user queries, then explore safer prefix matching, spelling variation, sparse-index acceleration, query batching, richer recommendation explanations, and more adaptive follow-up questions. The long-term goal is to make Dory better at understanding evolving customer intent while remaining grounded in catalogue text, transparent to debug, and inexpensive to run locally.

Built With

• python
• splade-v3-lexical