Inspiration

The problem statement calls out something every online shopper has felt: keyword search can't tell a browsing customer from a buying one, and it forgets what you told it two turns ago. Scout doesn't just search the catalog once, it scouts it fresh every turn, using whatever the conversation has revealed so far.

How we built it

We started by reading the evaluator's own source code before writing a line of agent logic — and that changed the whole design. The simulated "customer" isn't an LLM; it's a deterministic template engine that discloses facts from a hidden intent_card built directly from the target product's own catalog fields. One concrete consequence: asking about the other attribute always surfaces the next undisclosed constraint, regardless of its real type. Once we understood that, Scout's dialogue strategy became simple and reliable — ask the single most information-dense clarifying question every turn, track disclosed constraints as a persistent, decaying-on-override state, and re-rank as the state evolves.

The default agent is intentionally zero-dependency: SQLite's FTS5 extension does BM25 lexical retrieval, and a hand-tuned rerank layer combines that with structured slot matches (material, color, size, budget, use case). On top of that we built a full dense-retrieval layer — local MiniLM embeddings over the 50,000-product catalog, wired in with a verified automatic fallback to the pure lexical path if the embedding index is missing or a call fails.

Scoring, for reference:

$$TechnicalScore = 0.50 \times HitRate@10 + 0.30 \times MRR + 0.20 \times Efficiency$$

Baseline weak-BM25 starter agent: 0.107. Scout: 0.837.

Challenges we faced

The challenge was knowing when not to ship an idea. We tested and reverted two ranking hypotheses that sounded reasonable but measurably hurt the score once checked against the real evaluator: a "hard constraints are more reliable than soft ones" weighting scheme that turned out to rest on a false premise once we traced how the simulator actually builds those lists, and a tiered exact-match filter that was a complete no-op. And after building the full dense-retrieval pipeline, a 17-configuration ablation showed every way of blending in dense similarity net regressed the score — because this benchmark's disclosed constraints are literal substrings lifted straight from the target's own listing, so lexical matching is already close to ground truth for nearly every session. We shipped the RAG pipeline calibrated to zero rather than force it active for the sake of using it: a documented negative result we can defend, instead of a feature we can't explain.

What we learned

The most valuable input to this project wasn't the product catalog — it was the evaluator's own code. Reading exactly how the simulated customer behaves turned "build an NLU system" into "build a system that asks maximally informative questions," which is a much smaller and more reliable problem.

We also learned that recall and ranking precision fail independently. A 94% hit rate looked like a solved problem until we checked where in the top 10 the correct answer actually landed — only 55% of hits were rank 1. Fixing that needed a better scoring function, not more retrieval, which is exactly why the IDF fix mattered more than the RAG layer did for this particular benchmark.

What's next

Narrower dense retrieval — injected only into recommendation slots the lexical pipeline leaves low-confidence, instead of competing head-to-head with literal matches — and a small learned reranker (logistic regression over the features we already compute at rerank time) calibrated against the labeled dev sessions, replacing the remaining hand-tuned weights.

Built with

python, sqlite, fts5, bm25, tf-idf, sentence-transformers, minilm, numpy, regex, json, rag, hybrid-search, conversational-ai, dialogue-state-tracking, information-retrieval, amazon-reviews-2023-dataset, git, github, claude-code