Track 4: Shopping Copilot — AI Conversational Search and Recommendations Track 4: Shopping Copilot: AI Conversational Search and Recommendations is effectively an advanced guessing game, similar to Akinator, but with a much smaller set of products. With a confirmed target each session, our model aims to guess it with the least amount of turns as possible, with the highest amount of confidence possible. To do this, we came up with a system where the agent first builds an initial belief over which products are most likely to be the target. Once the shopper responds, we take the relevant products and score them using all the evidence we have collected so far, and then convert those scores into probabilities. This gives the agent a sense of confidence in whether the product at the top is actually the right answer. If confidence is high, the agent is more willing to commit and show a single product at rank 1. If confidence is low, it keeps more possibilities open, and may use another turn to gather more information. Before deciding what question to ask next, the agent uses our simulated shopper to play out different possible futures. It samples around $18$ likely target hypotheses and runs roughly $175$ Monte Carlo rollouts per turn, testing what could happen if it asked different questions. From those simulations, it estimates which question is most likely to improve the final result. Once the real shopper answers that question, the agent updates everything again: it re-scores the products, converts those scores back into probabilities, recalculates its confidence, and decides whether it is ready to commit or whether another question is still worth asking. Results With this solution, we have achieved the following scores: { "sample_count": 200, "hit_rate_at_10": 0.995, "mrr": 0.980208, "mttc": 2.66, "efficiency": 0.834, "recommended_technical_score": 0.958362, "reported_token_usage": { "prompt_tokens": 0, "completion_tokens": 0, "total_tokens": 0 }, "scenario_metrics": { "boundary": { "sample_count": 10, "hit_rate_at_10": 1.0, "mrr": 0.9125, "mttc": 3.0 }, "browsing": { "sample_count": 80, "hit_rate_at_10": 1.0, "mrr": 0.989583, "mttc": 2.65 }, "buying": { "sample_count": 80, "hit_rate_at_10": 0.9875, "mrr": 0.98125, "mttc": 2.1 }, "intent_override": { "sample_count": 30, "hit_rate_at_10": 1.0, "mrr": 0.975, "mttc": 4.066667 } } } The four key assumptions This agent is built on 4 key assumptions:
- The shopper's words come from the product's own catalogue text The requirements the harness gives shoppers are lifted from product feature and detail fields. So this is an inversion problem: given text produced from a row, find the row. Hence it is not a language-understanding problem.
- Rare wording identifies a product almost by itself $90.7\%$ of product specification strings occur on exactly one row; $95.6\%$ of products carry at least one such unique string.
- Targets are not drawn evenly from the catalogue Median review count is $12$ across the catalogue and $6{,}846$ across hidden targets. A product with $1000+$ reviews is $\sim 24\times$ more likely to be someone's target.
- Our reading of the catalogue is incomplete, and always will be We manage to record a size for only $19\%$ of products, a style for $26\%$. This single fact dictates that requirements may only ever add points, never delete candidates. Startup: preparing the catalogue Before the agent meets any shopper, it spends around 40 seconds going through all 50,000 products and turning the catalogue into something it can search really quickly later on. The full-text search index The first thing we build is a full-text search index. We take every product's title, category, features, details, store name and description and load all of that into an in-memory SQLite FTS5 table using BM25 ranking. At a high level, BM25 ranks products based on how closely their wording matches the shopper's message, while giving more weight to words that are rarer and therefore more informative. We also weigh certain fields differently, so for example a title match is worth much more than the same word appearing somewhere deep in the description. Keeping everything in memory means we can do this in milliseconds without relying on disk access, network calls or any external service. The structured attribute view Alongside that, we build a more structured view of the catalogue. For every product, we record things like material, colour, size, style, use case, features, category, brand and price. We do this using fixed vocabularies, for example a known set of materials, colours and features, combined with patterns for things like sizes and prices. We then store the result as an inverted index, so instead of asking, "What attributes does this product have?", we can very quickly ask things like, "Which products are cotton?" or "Which products are black?" We chose this dictionary-based approach instead of training a learned attribute extractor because we only had around 200 example conversations and no proper labelled dataset for product attributes. So rather than adding another model that could introduce uncertainty, we used something simple, deterministic and easy to understand. The specification fingerprint We then build what we call a specification fingerprint, which is one of the strongest pieces of evidence in the entire system. Every feature bullet and product detail line is turned into a lookup key pointing back to the products that contain it. We normalise the text and keep up to the first 120 characters, which means even if the shopper only repeats part of a specification, we can still match it. The important part is that we score these phrases based on how rare they are. Conceptually, we use: \text{rarity}(p) = \log!\left(\frac{\text{total products}}{\text{products containing that phrase}}\right) So if a phrase appears on thousands of products, we mostly ignore it. But if it only appears in one or two, it becomes extremely valuable evidence. This works because around $90.7\%$ of specification strings are unique to a single product. So if a shopper repeats wording from the actual product page, they have almost pointed directly at the answer without giving us the product ID. That makes specification fingerprints one of the most decisive signals we have. Category paths, coverage and fragmentation Finally, we also build a structured representation of the product taxonomy. Instead of treating category words separately, we keep meaningful phrases like "novelty socks" together. That is much more useful than looking at "novelty" and "socks" as two unrelated words. From these attributes, we also calculate two important statistics: coverage and fragmentation. Coverage tells us how often we successfully recorded a particular attribute across the catalogue. Fragmentation tells us how many different possible values that attribute has and how difficult it is to match reliably. The category phrase exists because "novelty socks" as a unit selects a median of $166$ products, whereas "novelty" or "socks" as loose words selects thousands. Coverage and fragmentation are later used by the imagined shopper to judge how likely a question is to produce a usable answer. These become important later when we build the simulated shopper, because they let the agent understand not only what information exists in the catalogue, but also where its own understanding of the catalogue is weak. Candidate generation and scoring Now that the agent has finished preparing the catalogue and building the simulated shopper, it receives the user's first real message and starts turning that message into a ranked set of possible products. Candidate generation The first step is candidate generation. Instead of scoring all 50,000 products every turn, which would be wasteful, we build a smaller candidate pool from four different sources and then keep the best $512$. BM25 text search — the top $3{,}000$ products, which gives us the products that most closely match the user's wording. Quoted specification matches — any product matched by a quoted specification. This is important because a rare phrase from a product's feature text can be much stronger evidence than general text similarity, so those products are injected directly even if BM25 did not rank them highly. Category — every product under the category the user mentioned. This helps us recover products that may be relevant even when their category words are too common to stand out in normal text search. Popularity backfill — if the candidate pool is still too small, we use a popularity backfill to make sure we still have enough plausible products to reason over. Anything that has already been shown to the user is removed. The reasoning here is simple: if one of those products had been the hidden target, the session would already have ended, so previously shown products are effectively free negative information. In practice, we score around $3{,}000$ products on a typical turn, with the largest candidate pool we measured being just over $4{,}100$. This gives us a good balance between recall and speed: we avoid repeatedly scoring the entire catalogue, but we still have multiple retrieval routes that can rescue a product if ordinary text search misses it. The score Once we have that candidate set, every product receives a single score made up of 13 different contributions. The easiest way to think about this score is as a log-posterior. In simple terms, every product starts with an initial belief, and then each new piece of evidence from the shopper either strengthens or weakens that belief. Structurally, this is similar to Naive Bayes, where different pieces of evidence are added together in log space: \log P(\text{product} \mid \text{evidence}) \;\propto\; \underbrace{\log P(\text{product})}{\text{starting belief}} \;+\; \sum{i} \log P(e_i \mid \text{product}) The starting belief includes things we found were predictive before even considering the conversation, such as: how many reviews the product has, whether it has a price recorded, and how many feature bullets it contains. Then we add the evidence collected from the conversation, including: quoted specification matches, category evidence, the structured attributes we extracted, the quality of the normal text match, and the short phrase matches we remembered from the shopper's wording. So rather than one signal deciding the answer, the final score is built by accumulating lots of smaller pieces of evidence. For example, two products might both be in the right category and both match the requested material. At that point they may look almost identical. But one of them might have much stronger review-count evidence, have complete price information, and match an exact phrase such as "machine washable". What the log-posterior view exposed One thing that was really useful about thinking of the score as a log-posterior is that it helped us find problems in our own implementation. At one point, three of the prior terms were disabled because they had been labelled as "redundant". But when we looked at the scoring system properly as independent pieces of evidence, we realised they were not redundant at all. Turning them back on was worth around $0.020$ in score. We found a similar issue with categories. Originally, a category like "novelty socks" was being broken down into loose individual words. Once we treated the category itself as its own piece of evidence, rather than just more text, that was worth another roughly $0.010$. So the main idea here is that by the end of this stage, we have gone from roughly 50,000 products to a focused candidate set, and every candidate now has one score representing everything we currently know about how likely it is to be the product the shopper has in mind. That score is what we use in the next stage, where we turn these raw values into actual probabilities to gauge how confident we are that the product currently at rank one is really the answer. Without the phrase row, A still leads — but almost entirely on popularity, which is a guess. With it, the margin comes from what the shopper actually said. That is the difference between a lucky lead and a justified one. Confidence: turning scores into honest probabilities In order to turn the scores into honest probabilities, we opted for a probability distribution converted using Softmax. We exponentiate each score, divide by the total: P(i) = \frac{\exp(s_i / \tau)}{\sum_{j} \exp(s_j / \tau)}, \qquad \tau = 0.75 A temperature of $\tau = 0.75$ sets the sharpness: a gap of $0.75$ points means roughly two-to-one odds. A small floor ($0.02$) is mixed in so belief can never collapse entirely onto one row. The old version assumed position 1 was always equally trustworthy, giving it identical treatment for a product backed by a unique quoted spec and for the arbitrary winner of a thousand-way tie. Since confidence is what decides how many products to show, that decision was being made blind. This is the second-largest component in the system, worth $0.049$. We did not choose a temperature of $1$ as the best fitting model was not always the best deciding model. Fitting the temperature by maximum likelihood gives $\tau = 1.0$ and scores $0.910$. Fitting it for the decision gives $\tau = 0.75$ and scores $0.922$. Likelihood cares about the whole 512-wide distribution; the policy only ever asks how much belief sits on the leader. Planning: the Monte Carlo simulation Now that we have a ranking of the candidate products in mind, we need to ask the next question. In order to determine what is the best question it could ask, we did a Monte Carlo Simulation using $18$ simulations of distinct candidate products, with each treated as the supposed answer. $32$ draws weighted by the belief from Act 6; duplicates collapse to $\sim 18$ distinct hypotheses. These are called particles, borrowed from particle filtering. Reasoning about all $512$ possibilities at every future turn is far too expensive. Sampling in proportion to belief spends effort where it matters — likely candidates get imagined more often. Filtering the question set We filter the ten possible questions down to those worth simulating. How. Drop anything the shopper already refused, anything asked twice, and anything no surviving candidate even has a value for. Why. If no remaining product has a recorded size, no answer about size can separate them, so asking wastes a turn regardless of how good the question sounds in the abstract. The rollout For each allowed question and each hypothesis, simulate up to four more turns and record what the session would have scored. This is done using a rollout: Ask the question, let the imagined shopper answer, narrow the candidates, check whether the hypothesis would now be shown. If not, remove what would have been published and continue. After four turns, estimate the remainder with a learned value function rather than simulating further. Why. Four turns deep because the average session ends in $2.7$ — simulating further mostly adds compute and noise. Truncating with a value estimate rather than scoring zero avoids systematically undervaluing longer paths. The following is an example of our monte carlo simulation: The quarantined evaluator shortcut We discovered the competition's simulated customer answers the open-ended "anything else?" question by reading out the product's own description word for word. Exploiting that scores higher. We built it, measured it at $0.964$, and quarantined it behind a switch that is off by default — it is a quirk of one test harness, not a fact about shoppers. The gap between honest and exploit is now just $0.005$, and both find the same products. Deciding Choosing the question Average each question's simulated outcomes, blend with the rehearsed instinct, take the best. V(q) = 0.75 \cdot V_{\text{live}}(q) + 0.25 \cdot V_{\text{prior}}(q) $75\%$ live simulation, $25\%$ the prior from the start. Then a final check: is asking anything better than staying silent? Silence must win by a clear margin before it is chosen. The silence margin exists because staying quiet forfeits a whole turn of information, and premature silence is far more costly than a mediocre question. The bias is deliberately asymmetric. Choosing how many products to show Decide between publishing one product and publishing ten — nothing in between. How. For each option: the chance the target is in that list (converting now, at whatever rank it lands) versus the value of continuing with a fresh turn. The first part is computed exactly; the second reuses the same rollouts. Highest expected score wins. Why. The session ends the instant the target appears, so ten products is ten simultaneous guesses — if the right one is seventh, you are scored seventh. Showing one means being right scores perfectly, and being wrong buys another turn. Worth $0.082$, more than everything else combined. Publishing four is hedging: it converts at a middling rank, which the scoring rule never rewards. We measured $2$, $3$ and $5$ on the menu and they cost $0.010$. Either name the product or buy another turn — the middle has no value. The loop continues So after it asks the question, the loop repeats identically with the following 3 exceptions: Evidence accumulates. Requirements are added, never replaced — a later mention of an unrelated colour must not overwrite the colour they actually asked for. The shown set grows. Every published product becomes a proven non-target and is permanently excluded. Confidence sharpens. As evidence stacks, the gap between first and second widens, and the agent's behaviour changes with it. Hence why we did not need any LLM. This problem is entirely solvable using logic and rarity. Summary of results Confidence and correctness track each other closely, which is the real test of the Act 6 machinery. When the agent commits to a single product it is usually because the gap has widened past the point where it has ever been wrong. When it stays broad, it genuinely doesn't know yet — and it says so by publishing ten. The one it never finds A single session fails, and it fails the same way in every version of this system we have ever built. The product simply never surfaces high enough — it stalls around 320th place even with nine requirements and a matching quoted specification. It is a retrieval limit, not a reasoning one, and it is worth $0.0025$. We have left it. How much is actually left The theoretical maximum on this task is $0.982$, because the harness forbids finding the answer early in certain scenario types. And if we could perfectly re-order the products we already retrieve — a measurement we ran — we would score $0.9922$. The target is in our shortlist on turn 1 in every single session. Retrieval is finished; all remaining headroom is ordering. Safety nets Planning can fail without costing a session. The whole planner is wrapped so that any fault falls back to a simple heuristic — pick the question that splits the candidates most evenly, weighted by how answerable it is — and publishes the full ten. A crash degrades the turn; it never loses the session. The last turn always shows ten. On turn 10 there is no next turn to buy, so holding back has no value and the widest list is always correct. Evidence never deletes. Assumption four is enforced everywhere, including inside the simulation. Belief never fully collapses. The small floor in the softmax means no amount of evidence makes the agent certain. Questions retire, permanently. A refusal removes that question for the rest of the session, so a turn is never wasted twice the same way. From Wordle to All In: The Story of My TechJam Shopping Copilot I started this project with what seemed like a search problem. A shopper would describe what they wanted, and my system would search a catalog of 50,000 products. The target product had to appear in the Top 10 within ten conversational turns. The score rewarded three things: finding the target, ranking it highly, and finding it early. At first, the obvious solution was retrieval: collect the shopper's words, run BM25, and return the most relevant products. That approach scored only $0.106710$. It was the first and most important reality check. The problem was not simply: "Which product contains these words?" It was: "Which question should I ask next so that thousands of plausible products become a small, identifiable set?" That realization led to the first major inspiration behind the project: Wordle. The Wordle idea Wordle is not solved by guessing random words. A good guess is valuable because it divides the remaining possibility space. Even if it is not the final answer, it reveals information that makes the next guess better. I applied the same idea to shopping. Instead of asking whatever attribute sounded natural, the copilot examined its current candidate set and asked: Which unresolved attribute most evenly divides the remaining products? If 100 candidates remained, a weak question might separate them into groups of 90 and 10. A stronger question might separate them into groups of 52 and 48. The second question reduces uncertainty much more effectively. This became the candidate-splitting strategy: \text{question value} = \text{coverage} \times \left(1 - \frac{\text{largest answer bucket}}{\text{usable candidates}}\right) The system could ask about material, color, size, style, brand, budget, features, or use case. It avoided attributes already answered, declined, or insufficiently represented in the catalog. This Wordle-inspired change transformed the project. The score jumped from roughly $0.410$ without clarification to $0.727529$ with candidate-splitting questions. That was the moment I understood that the copilot's most powerful capability was not recommending products. It was actively acquiring information. The first lesson: conversation is part of retrieval Before the Wordle technique could work, the system needed memory. The original baseline searched only the latest message. If a shopper said: Turn 1: I need a jacket. Turn 2: Preferably waterproof. Turn 3: Black would be good. the third search could forget that the shopper wanted a waterproof jacket. I therefore added cumulative conversation history. That more than doubled the baseline performance. But memory introduced a new problem: old information is not always valid. A shopper might say: Actually, I changed my mind. Show me boots instead. Blindly retaining the old jacket constraints would poison retrieval. The solution was override-aware state: keep history within the current intent, detect explicit changes of mind, deactivate stale constraints, and allow previously shown products to become eligible again. This taught me that conversational search requires two opposing abilities: remembering useful evidence; knowing when to let it go. Exploration without repetition The next inspiration came from search and recommendation systems: exploitation alone is not enough. Returning the same ten highly ranked products repeatedly wastes turns. I introduced hard repeat suppression, retrieving a larger candidate pool and excluding products already shown during the current intent. This dramatically increased catalog coverage and improved every aggregate metric. The system was now behaving less like a static search engine and more like a search process: remember evidence → recommend the strongest candidates → avoid wasting future turns on the same failures → ask a discriminative question → update the search space The override reset was essential. Without it, exploration would become permanent exclusion and could hide the correct product after the shopper changed direction. Not every preference is a requirement The next challenge was language strength. These statements should not have equal force: It must be leather. I want leather. I would prefer leather. Maybe leather. I do not care about the material. I introduced confidence-weighted constraints and later refined them into hard, normal, and soft evidence. Hard requirements received strong positive support when matched and strong penalties for explicit violations. Preferences influenced ranking without behaving like absolute filters. No-preference answers contributed no constraint. This improved ranking and convergence, but the deeper learning was more valuable: Natural language does not merely contain attributes. It contains degrees of commitment. The agent needed to model both what the shopper said and how strongly they said it. Why the clever question formulas failed After the Wordle strategy worked, I tried to make it more sophisticated. I tested: ordinary entropy; probability-weighted entropy; expected candidate elimination; predicted competition-score improvement; probability-weighted candidate splitting; minimum-confidence question gating. These ideas were theoretically attractive. In practice, none beat the simple largest-bucket split. That was one of the most useful stretches of the project. Entropy rewarded a broad distribution of possible answers, but the most informative-looking question was not always the most useful question. Expected-value formulas depended on probabilities that were only rough rank-based approximations. Gating suppressed questions that looked weak numerically but still produced useful evidence. The lesson was: A more complicated mathematical objective is not automatically a better proxy for the real task. Simple candidate splitting worked because it controlled the worst remaining ambiguity. It was easy to interpret, deterministic, and closely connected to what happened after the answer. Answerability: a question is useful only if it can be answered A perfectly discriminative question has no value if the shopper — or simulator — usually responds with no preference. The diagnostics revealed enormous differences between attributes. Feature and material questions were usually answerable. Brand, budget, size, style, and use-case questions were often not. I therefore adjusted the Wordle score by empirical answerability: \text{useful question} = \text{candidate separation} \times \text{catalog coverage} \times P(\text{substantive answer}) This produced another major gain, reaching $0.769408$ on development. This became one of the central principles of the project: The value of a question is not how much it could reveal in theory. It is how much useful information it is likely to reveal in this interaction. The fingerprint inspiration The next bottleneck was representation. Ordinary lexical retrieval treated the conversation as a bag of words. But products often contain distinctive combinations of catalog phrases: waterproof leather upper fold-over clasp pull-on closure 100% cotton If the shopper revealed one of these phrases, it could behave almost like a fingerprint. This inspired the fingerprint retrieval route: build an index only from catalog-visible product phrases, then search the accumulated dialogue for exact or normalized phrase evidence. The important safety boundary was that the fingerprint index used catalog information only. It did not contain hidden target labels or private session information. The fingerprint approach improved robust performance and also became exceptionally powerful when combined with the evaluator's other response behavior. Discovering the "other" shortcut When I tested the allowed clarification attribute other, I discovered that the released evaluator responded with unusually detailed product-specific information. This changed the nature of the local problem. A normal question might reveal: material: leather An other question could reveal a much more discriminative phrase resembling catalog metadata. Combined with fingerprint retrieval, this often moved the target directly near the top. I progressively tested: always asking other; asking other first; repeating other until exhausted; one early other; two early other questions; using other only after named attributes were exhausted. The repeated wildcard strategy became E022, the released-evaluator champion. Its final full-set score was $0.855036$, with MRR $0.673454$ and MTTC $2.725$. But this success came with an important engineering and research decision: I refused to pretend that E022 was universally robust. Its advantage depended on how the released simulator interpreted other. A private evaluator or real shopper might not reveal the same detailed information. I therefore separated the project into two tracks: Track Behaviour Robust champion ordinary named questions only Evaluator champion repeated wildcard behavior That separation became one of the strongest parts of the project. It allowed me to report the highest local score without confusing benchmark exploitation with general conversational intelligence. The "all in" technique The robust system's later development was my "all in" phase. By this point, I no longer trusted any single source of evidence. The system needed to use everything it could legitimately observe: the current message; the full active conversation history; hard, normal, and soft constraints; intent overrides; declined attributes; previously shown products; BM25 lexical relevance; exact catalog phrases; controlled attribute values; category hierarchy; material evidence; feature evidence; title, details, description, and store metadata; structured field matches; answerability; candidate ambiguity; repeat suppression. This was not "all in" in the sense of assigning every feature a huge weight. It meant giving every credible evidence source a defined role. The aggregate field-aware retrieval system, E025D, combined phrase and structured evidence rather than forcing a choice between them. It improved robust performance to $0.808106$ on development and $0.824324$ on holdout. This was another key learning: Strong systems often come from combining complementary evidence routes, but fusion must remain disciplined. More routes do not automatically mean better results. I tested broader multi-route consensus, wider candidate pools, evidence-coverage reranking, structured BM25 expansion, protected reranking, and several fusion rules. Most regressed. The successful "all in" architecture was therefore not a maximal pile of techniques. It was the set of techniques that survived controlled ablation. The dead-turn discovery After field-aware retrieval, the robust system was already finding almost every target. Yet a few sessions still failed because the question policy became exhausted. The ordinary selector could decide that no question was strong enough, even though unresolved named attributes still had some useful candidate coverage. The system would continue recommending without acquiring new information. These were dead turns. E035B fixed this with a small but important rule: When ordinary question selection returns nothing, ask the highest-scored remaining named attribute with positive coverage. This was not a new question-scoring theory. It reused the existing evidence and removed an unnecessary gate. That simple exhaustion fallback improved the development score from $0.808106$ to $0.821151$ and held on holdout at $0.830124$. It became the final robust champion. The lesson was almost philosophical: Do not stop learning merely because the best available question is imperfect. When more retrieval stopped helping Once HR@10 reached approximately $0.985$, I investigated whether adding more candidates could recover the remaining misses. The candidate-union oracle showed only $0.005$ additional hit-rate headroom. Wider structured pools could admit one difficult target, but it still ranked around 100 rather than entering the Top 10. This separated two problems: target absent → retrieval problem
target present but buried → ranking or ambiguity problem Broad retrieval expansion no longer had enough value to justify its noise. The system had reached a point where finding more candidates was easier than knowing which of many plausible candidates was correct. The oracle that looked too good A rank oracle produced a technical score around $0.984$ by placing the target first whenever it was already in the candidate union. At first, that looked like enormous ranking headroom. But the oracle knew the hidden answer. The runtime system did not. I investigated high-ambiguity turns and compared each target with the competitors ranked above it. In $93.7\%$ of the relevant pairs, the competitor and target matched the same structured fields. Even after richer value and lexical comparisons, many remained observationally equivalent. The catalog might contain hundreds of products consistent with: category: wallet material: leather color: black If the target and competitors satisfy everything the shopper has revealed, a ranker cannot reliably know which hidden item is correct. This reframed the oracle: The oracle measured hidden-answer headroom, not necessarily learnable headroom. That distinction prevented me from chasing an unrealistic $0.98$ system through increasingly elaborate rerankers. Asking a better question versus building a better ranker The high-ambiguity question oracle showed that a better allowed question could sometimes improve reciprocal rank substantially. This suggested that the remaining MRR and MTTC opportunity was not just: "Rank the same evidence better." It was: "Acquire more discriminative evidence before ranking." However, the oracle chose questions using simulator-aware counterfactual answers. A real runtime policy needed to predict question value using observable information only. I tested ambiguity proxies, conditional specificity, collision-breaking questions, and catalog-trained interaction learning. The most promising result was category-conditional IDF. It improved development to $0.834609$, won all five cross-validation folds, and had a strongly positive bootstrap interval. Then it scored $0.829067$ on holdout, slightly below E035B's $0.830124$. I rejected it. That decision represents one of the project's most important methodological achievements. It would have been easy to promote the more sophisticated model because its development evidence looked excellent. The untouched holdout showed that the improvement did not generalize reliably enough. The lesson was: A convincing explanation, five winning folds, and a positive bootstrap interval still do not overrule the final holdout gate. Learning to stop Near the end, I tested almost every obvious remaining direction: new entropy rules; question gating; contradiction handling; soft-constraint decay; scenario routing; diversity; broader candidate admission; route fusion; field-coverage ranking; structured query expansion; conditional IDF; bounded wildcard schedules; ambiguity-triggered questions; collision-based question prediction; synthetic interaction reranking; wildcard and field-evidence factorial combinations. Most failed. Some were neutral. A few improved development but failed holdout. Several turned out to duplicate behavior already present. This was not wasted work. Each rejected branch reduced uncertainty about the architecture. By the end, E035B was not merely the best number in a table. It was the model that remained standing after repeated targeted attempts to replace each of its major subsystems. The final robustness lesson The last stress tests showed exactly where the robust champion deserves — and does not deserve — the word "robust." It handled: catalog capitalization; no-preference paraphrasing; override paraphrasing; ordinary conversational variation. It showed mild sensitivity to filler and secondary metadata. But it was materially sensitive to: punctuation removal; missing leaf categories; missing feature metadata. Removing feature metadata dropped the score from approximately $0.821$ to $0.436$. That result exposed a genuine structural dependency. The system is not merely helped by features; it relies heavily on them. If the competition uses the exact frozen catalog, that may be acceptable. If catalog completeness or formatting changes, it is the largest remaining technical risk. The final conclusion was therefore not simply: "The model is robust." It was: "The model is robust to conversational variation and casing, while intentionally depending on rich catalog structure — especially features and category depth." My final system The project ended with two frozen champions: Champion Role Full score Key metrics Dependency E035B Robust champion $0.823394$ HR@10 $0.985$ No wildcard dependency E022 Released-evaluator champion $0.855036$ MRR $0.673454$ Depends on repeated other semantics The full 30-test suite passes, and the final evaluation reproduced the archived champion results without regression. What this project ultimately taught me I began by thinking I was building a better product search engine. I ended up learning that the real challenge was information acquisition under uncertainty. The Wordle technique taught me to ask questions that divide the search space. Answerability taught me that theoretical information is useless if the user cannot provide it. Confidence-weighted state taught me that a preference is not the same as a requirement. Override handling taught me that memory must be reversible. Repeat suppression taught me that exploration needs structure. Fingerprint retrieval taught me that distinctive catalog phrases can behave like product identities. The other experiments taught me to distinguish evaluator optimization from generalizable behavior. The all-in architecture taught me to combine every legitimate source of evidence without assuming that every additional mechanism helps. The oracle experiments taught me the difference between theoretical and learnable headroom. The failed rerankers taught me that complexity needs causal evidence. The holdout rejection of E046 taught me to trust experimental discipline even when a result looks exciting. And E035B taught me that sometimes the final meaningful improvement is not a grand new model. It is recognizing that the system stopped asking questions too early. The conclusive story The journey moved through three ideas: Search → ask better questions → understand the limits of what can be known The original BM25 system tried to retrieve the answer directly. The Wordle system learned to reduce uncertainty one question at a time. The all-in system combined dialogue state, catalog evidence, exploration, ranking, and clarification into one coordinated search process. The final research phase discovered that the remaining failures were no longer dominated by one obvious algorithmic mistake. They were increasingly caused by missing retrieval evidence or by multiple products being indistinguishable under everything the shopper had revealed. That is why the strongest conclusion is not: "I built the mathematically best possible shopping agent." It is: I systematically identified and removed the dominant conversational-search bottlenecks — from memory and exploration, through Wordle-style clarification, to all-in evidence retrieval — until the remaining errors were largely bounded by the information available to the system. The final model is valuable not only because it scores well, but because I know why it works, where it fails, which tempting alternatives were disproven, and what new evidence would be required to improve it further.
Log in or sign up for Devpost to join the conversation.