Main teaser
Flow chart

Seekly: Your shopaholic Bestie

Project summary

Seekly is a multi-turn shopping agent that searches a frozen catalog of 50,000 Amazon products. It accumulates constraints across a conversation, asks targeted clarification questions, handles preference overrides, and progressively opens its recommendation list as the evidence improves.

The shipped path is deterministic and fully offline. On the 200-session public development set, it finds all 200 targets, places 190 at rank 1, and achieves Hit Rate@10 1.000, MRR 0.965853, MTTC 2.305, and a technical score of 0.963656.

Inspiration

The obvious way to build a shopping agent is to make it a better search engine: retrieve more candidates, rank them more intelligently, and answer faster. We started there. Then we read the evaluator and realized that the harder problem was deciding when - and how broadly- to recommend.

The session ends as soon as the target appears anywhere in the Top 10. A target at rank 9 ends the conversation just as permanently as a target at rank 1, locking in a reciprocal rank of 0.111 with no chance to improve it on a later turn. Our early agent often had the right product in its candidate pool but exposed too many weakly supported alternatives too soon.

That changed the project. Instead of treating every turn as a request for a full result page, we designed the agent around evidence-controlled disclosure. It asks when the request is under-specified, shows only the strongest candidate during the first two turns, and opens the full Top 10 once more evidence has arrived. The key question is no longer only “What should we rank first?” but also “How much of this ranking should the customer see now?”

What it does

Seekly turns a shopping conversation into a structured, evolving search state.

Every requirement is stored with its value, the turn when it arrived, and its provenance: volunteered by the customer, supplied as an answer, or introduced by an intent override. This distinction matters. When a customer says “ignore my earlier preference,” the agent retracts the preferences they volunteered under the old intent while preserving factual answers they gave to the agent’s questions.

On each turn, the agent:

interprets the new message with deterministic rules and an optional LLM fallback for unclear natural language;
updates category, constraints, rejected attributes, buying/browsing intent, profile signals, and override state;
retrieves and fuses candidates from exact and lexical routes, with optional dense semantic routes;
applies category, historical-context, and post-override provenance signals;
reduces repeated exposure without permanently deleting previously shown products;
asks the next most useful clarification question; and
returns one high-confidence candidate on turns 1–2, then up to ten candidates from turn 3 onward.

If the user has no more information to provide, the system stops repeating questions and switches to facet-diverse exploration. This helps low-popularity or metadata-ambiguous products surface across later turns instead of allowing the same ranking head to occupy every recommendation slot.

Buying, browsing, and post-override conversations use different retrieval weight profiles. Natural-language questions remain compatible with the evaluator’s structured ask_attribute contract, and every remote-model failure falls back to the deterministic offline path.

How we built it

We built the agent as a stateful Python pipeline over the same frozen 50,000-product catalog used by the evaluator.

Conversation and policy layer

The conversation layer combines a regex-first message parser, optional GPT-5.6 Luna extraction and intent classification, structured constraint memory, profile distillation, rejection tracking, and explicit override semantics. The clarification policy considers what the customer has already said, what they declined to answer, which dimensions their profile suggests they care about, and which attributes best separate the current candidates.

The recommendation policy is progressive rather than fully silent. During turns 1–2, the agent continues gathering evidence but exposes only the current rank-1 candidate. From turn 3 onward it can return the full Top 10. Previously exposed products receive a recoverable rank-space penalty, allowing fresh candidates to surface while still letting new evidence pull an earlier product back to the top. An intent override resets exposure history because products shown for the old request are not negative evidence for the new one.

Retrieval and ranking layer

The deterministic retrieval path uses exact constraint intersection and fielded BM25 over title, category, features, and store metadata. Multiple rankings are combined using weighted Reciprocal Rank Fusion, so routes contribute through rank position rather than incompatible raw score scales.

We also built two optional Qwen3-Embedding-8B views of every catalog item:

an identity view describing what the product is; and
an attribute view describing the properties it has.

Each view is stored as a 50,000 × 512 float16 matrix. At runtime, dense search is filtered to the strongest structured candidates and, by default, gated to post-override turns. We also implemented GPT-5.6 Luna query rewriting and Qwen3-Reranker-8B cross-encoder reranking as controlled ablations. They remain off in the shipped configuration because they did not outperform the offline path.

After fusion, the system applies exact taxonomy-suffix evidence, a weak retracted-context route, post-override catalog-provenance likelihood, exposure-aware demotion, and exhausted-state facet exploration. A local dashboard can replay every conversation turn and show aggregate metrics and retrieval diagnostics.

Evaluation discipline

We treated every feature as an experiment. Each configuration was compared with a forced-off control, and the baseline had to reproduce exactly before we trusted a claimed gain. The repository now contains 79 automated tests covering conversation state, override erasure, intent routing, constraint locking, dynamic truncation, suppression, progressive recommendations, exposure recovery, semantic-index behavior, dashboard replay, and evaluator correctness.

In addition to the 200 public sessions, we generated a deterministic 100-case diagnostic set whose targets do not overlap the public targets. It preserves the official scenario proportions and includes deliberately ambiguous products. The current evaluated diagnostic configuration finds 99 of 100 targets; all 83 cases whose disclosed evidence narrows the catalog to at most ten matching products are found. We use this set as a generalization diagnostic, not as an estimate of the private leaderboard.

Challenges we ran into

The evaluator rewards controlled exposure

Our first major challenge was counterintuitive: returning more relevant products could reduce the score. Once any target appears in the Top 10, the session stops, so a broad early list can convert too soon at a poor rank. Fully withholding recommendations improved MRR but hurt time to conversion. The final policy resolves that trade-off by returning one candidate on the first two turns and widening later.

Preference overrides require provenance, not just recency

A naive override implementation deletes everything learned before the override. That is wrong when the customer previously supplied factual answers in response to our questions. We had to model where each constraint came from, retract only volunteered preferences, preserve answer-derived facts, and separately reset recommendation exposure for the new intent.

Some failures are information failures, not retrieval failures

The original public miss was a “Grandma” long-sleeve novelty shirt. The target was already in the candidate pool, but the evaluator disclosed only generic properties such as cotton, grey, and imported. It never revealed the words that uniquely identified the product. No reranker can reconstruct evidence the conversation never supplies. We recovered the case through facet-diverse exploration rather than a target-specific rule.

The diagnostic set exposed the same issue in a different form: one leather belt remained tied inside a group of 18 products with indistinguishable disclosed metadata. That failure showed us that the next improvement is a better exploitation-versus-diversity allocation inside tied groups, not another generic semantic model.

Our biggest planned model made the system worse

We implemented semantic reranking because it looked like an obvious improvement. Against an already strong lexical candidate set, however, the reranker moved the correct product down more often than up and could push targets out of the scored Top 10. A previous 25:1 blend had appeared safe only because it was mathematically too weak to change the output. Giving the model enough authority to matter revealed the regression.

Unrestricted dense retrieval had a similar problem. It helped a narrow post-override slice slightly, but applying it to every conversation lowered the public score. The final offline configuration therefore does not depend on an API, while the semantic components remain available as measured, gated experiments.

Accomplishments that we're proud of

Improved the released BM25 starter from Hit Rate@10 0.125, MRR 0.068034, and MTTC 9.810 to Hit Rate@10 1.000, MRR 0.965853, and MTTC 2.305 on the public development set.
Found all 200 public targets and placed 190 of them at rank 1 with a deterministic, zero-token runtime path.
Recovered the only originally missed public case without editing the evaluator or hard-coding a product ID.
Built explicit constraint provenance and correct multi-override behavior instead of treating conversation memory as an undifferentiated text blob.
Turned a fully silent early policy into progressive disclosure, improving both reciprocal rank and time to conversion.
Built optional two-view dense retrieval, query rewriting, and reranking, then had the discipline to leave them disabled when controlled experiments showed they did not improve the shipped system.
Added a reproducible 100-case unseen-target diagnostic suite and 79 automated tests.
Built a local replay dashboard that makes session-level ranking behavior inspectable rather than relying only on one aggregate score.

What we learned

Read the evaluator before optimizing the model. Our largest gains came from understanding the stopping rule and recommendation contract, not from adding a larger neural component.

Ranking quality and recommendation policy are different problems. A system can have an excellent candidate order and still score badly by exposing too much of it too early. Conversely, hiding everything can protect rank but waste turns. The best policy controls the width of the visible list as evidence changes.

Conversation memory needs provenance. “What the user originally wanted,” “what the user told us when asked,” and “what the user wants now” cannot safely be collapsed into one bag of keywords.

Negative results are engineering results. Query rewriting, dense retrieval, reranking, slot decay, dynamic truncation, and constraint locking were not accepted because they sounded useful; they were measured against forced-off controls. Some were neutral, some were useful only in a narrow route, and some were harmful. Removing or gating them made the final agent stronger, cheaper, and easier to reproduce.

Finally, models cannot recover information that was never disclosed. When several products are identical under all observable evidence, the correct response is calibrated exploration and transparent uncertainty—not invented confidence.

What's next for Seekly

The highest-priority next step is robustness to genuine natural-language paraphrase. The parser is regex-first with an optional LLM fallback, but the fallback needs a much broader adversarial evaluation set covering colloquial answers, negation, corrections, and multi-constraint replies.

We also want to improve exploration inside metadata-tied groups. The diagnostic miss suggests reserving part of each later Top 10 for the nearest unseen candidates while keeping part for cross-facet diversity. This should be validated against both the public set and the unseen-target diagnostic suite before shipping.

Dense retrieval will remain experimental until it demonstrates unique, repeatable contribution beyond the structured candidate set. Rather than increasing its global weight, we will test confidence-based routing for the narrow cases where lexical evidence is genuinely weak.

Longer term, we want profile learning to update across sessions, distinguish stable preferences from one-off needs, and help choose clarification questions without overriding the customer's current request. We also plan to evaluate the agent with real shoppers, where open-ended answers can provide the distinguishing evidence that a fixed simulator cannot.