techjammers

Inspiration

Traditional e-commerce search treats every query similarly. However, someone typing “black leather boots under $80” has very different needs from someone saying “I’m looking for boots, but I’m still browsing.” One shopper wants immediate precision; the other needs broader exploration and useful guidance. We wanted to discover how far a fully offline conversational search system—without an external LLM API—could go in adapting its retrieval strategy turn by turn. The challenge was to identify the shopper’s hidden target product from a catalog of 50,000 items within a strict 10-turn conversation budget.

What it does

AgentV4 is an offline conversational product-search agent. On every turn, it: Updates its memory with the shopper’s latest preferences. Detects special situations such as an intent change or an uninformative response. Searches the catalog using BM25 keyword retrieval. Activates semantic search and Reciprocal Rank Fusion when additional recovery is needed. Re-ranks products using the preferences collected during the conversation. Chooses the clarifying question expected to eliminate the most candidates. Returns a ranked product shortlist alongside the question. The agent does not wait until it has perfect information before recommending something. It continually exposes its current best guesses, allowing it to score an early hit while still gathering information. When confidence is low and the next question is highly informative, AgentV4 can return a smaller four-item shortlist. Otherwise, it returns up to the requested Top-10.

How we built it

How we built it AgentV4 is implemented as one standalone Python file containing three logical layers, all operating locally and without external API calls.

Hybrid retrieval At startup, the agent loads all 50,000 products into an in-memory SQLite FTS5 index. BM25 searches across: Product title Category Features Product details Brand or store Description The fields are weighted differently, with the title and category receiving the strongest influence. The same catalog is also converted into TF-IDF vectors using unigrams and bigrams. Truncated SVD compresses these sparse vectors into a 128-dimensional semantic representation. This allows the system to retrieve related products even when the shopper’s wording does not exactly match the catalog text. For ordinary buying and browsing conversations, AgentV4 preserves BM25’s precise ordering. When it detects an intent override or boundary-recovery situation, it combines BM25 and semantic rankings using Reciprocal Rank Fusion. This targeted fallback adds semantic recall without allowing semantic results to disrupt every normal search.
Conversational state tracking The agent maintains separate memory for every session, including: Active conversation messages Retired messages from an earlier intent Known attribute constraints Previously asked attributes The pending question The original product-category request Deferred questions Boundary and intent-override recovery state When the shopper says something such as “actually,” “instead,” or “I changed my mind,” AgentV4 retires the obsolete preferences while preserving the original product type where appropriate. This prevents contradictory constraints from contaminating the new search. It also distinguishes between: A genuine preference answer No additional preference A temporary refusal to answer An uninformative response An explicit change of intent
Information-gain question selection AgentV4 estimates possible answers for seven askable attributes: Material Color Size Style Budget Feature Use case For each unasked attribute, it examines the current candidate pool and calculates: question score = attribute coverage × answer-distribution entropy Coverage measures how many candidates contain usable information for that attribute. Entropy measures how effectively the possible answers divide the candidate pool. An attribute receives a high score when it is available for many candidates and produces a diverse answer distribution. This means the question is likely to eliminate a substantial part of the search space. The selected answer is then stored as a constraint. Products matching rare and therefore more discriminative answers receive a stronger reranking boost.
Confidence-aware output AgentV4 measures the separation between the highest-ranked products and the rest of the Top-10. It combines this ranking margin with the expected strength of the next question. If the ranking is uncertain but the question is highly informative, it returns a focused shortlist while gathering more evidence. During boundary recovery or immediately after an intent override, it expands retrieval from 100 to 300 candidates and returns the full Top-10 to maximize recall. Every design decision was tested using the organizer’s evaluator/local_evaluator.py on the 200-session public development set rather than being tuned only through intuition.

Challenges we ran into

Recall vs. narrow querying. An early version searched only on extracted keywords instead of the full conversation, and scored worse than the weak baseline (0.05 vs. 0.125 Hit Rate). This was a hard lesson learnt that narrowing information too early kills recall. Asking the wrong question. We found that ~93% of a weak scenario bucket's misses had the correct product already inside our retrieval pool, just outside the visible top 10. This means the bottleneck was which question we asked, not how hard we searched. A stonewall isn't a "no." In one scenario type, the very first clarifying question of a session always gets a non-answer, regardless of what's asked. Treating that the same as a genuine "no such preference" answer meant permanently losing the one piece of information that mattered most. This was fixed by learning to distinguish "no signal given" from "confirmed negative."

Accomplishments that we're proud of

AgentV4 achieved the following results on the complete 200-session public development set: Technical score: 0.738287 Hit Rate@10: 86% MRR: 0.566956 Mean turns to conversion: 4.09 Efficiency: 0.691 Reported external token usage: 0

What we learned

Conversational search is not simply repeated keyword search. Retrieval, dialogue state, question selection, and ranking have to be designed as one feedback loop. Our main lessons were: High recall is only useful if reranking can move the target into the visible results. MRR depends on product ordering, not merely whether the target enters the Top-10. A question should be selected based on the current candidate pool, not from a fixed attribute order. More retrieval routes do not automatically improve performance; poorly calibrated fusion can lower MRR. Semantic search works best as a targeted recovery mechanism rather than a universal replacement for lexical search. Intent changes require selective forgetting rather than either remembering everything or resetting everything. Non-answers, negative preferences, and intent changes must be represented as different dialogue states. Optimizing a single scenario can reduce the overall score, so every change must be evaluated across the complete dataset. Most importantly, we learned that a relatively small and explainable offline system can demonstrate surprisingly capable conversational-search behavior when its retrieval and dialogue strategies are adapted carefully.

What's next for techjammers

Our next step is adaptive candidate depth. Instead of always retrieving 100 products—or 300 during recovery—the agent could dynamically expand or contract the pool according to ranking confidence, the number of known constraints, and the expected value of the next question.