The Elimination Engine

Inspiration

The challenge looked semantic at first glance: a "conversational search" agent should need embeddings, an LLM reranker, maybe a vector DB. But a close read of the customer simulator showed something different: every reveal is a phrase lifted verbatim from the target product's own catalog text, and even the opening message embeds the evaluator's own category function output verbatim. That single observation reframed the whole problem. This isn't semantic retrieval, it's elimination. We got interested in how far a deterministic, purely lexical agent could go if we took that seriously instead of reaching for an LLM by default.

What it does

The Elimination Engine plays a multi-turn shopping assistant. Each turn it reads a short customer message, decides whether to ask a clarifying question, and returns up to 10 ranked parent_asin catalog matches, trying to surface the customer's hidden target product as early and as high-ranked as possible, within a 10-turn budget.

$$\text{TechnicalScore} = 0.50 \times \text{HitRate@10} + 0.30 \times \text{MRR} + 0.20 \times \text{Efficiency}$$

The shipped BM25 starter scores 0.1067 on the same public set. For scale, that's roughly a 9x composite improvement.

How we built it

agent.py is a thin scenario router over two independent retrieval tracks, each a complete agent in its own right:

• elimination/ — accumulates every disclosed phrase across the session as a hard exact-phrase filter, ranks what's left, and falls back to a diversified browse track on constraint-free turns. Wins Buying, Browsing, and Boundary scenarios, and generalizes best.
• lexical/ — a coarse-category bucket pre-filter feeding a two-tier BM25 index (SQLite FTS5), then a single linear reranker with an exposure window. Wins Intent Override, where a "replace my earlier preference" turn is nearly a no-op and hard-filtering can accidentally carve the real target out of the pool.

We didn't guess which track should own which scenario and measured instead. Each track was run standalone against an 800-session held-out set never tuned against, and we routed each scenario to whichever track scored higher on it. That netted a small but clean, zero-regression gain over either track alone.

Why the split isn't arbitrary. The two tracks treat a disclosed constraint completely differently. Elimination compiles constraints into a sequential AND filter, most-specific first — a phrase is kept only if it doesn't zero the whole pool, and once a product is filtered out, no later ranking step can bring it back. Lexical instead files each constraint into an attribute slot and only ever adds score; nothing is ever removed from candidacy. That difference decides the routing:

• Buying / Browsing / Boundary are purely additive — every turn discloses something new, nothing is ever retracted, so elimination's hard filter only ever narrows correctly, and its irreversibility is a feature: it aggressively drops the hundreds of irrelevant same-bucket neighbors that lexical's softer scoring still has to out-rank rather than exclude. Elimination also runs a dedicated diversified browse pass for the constraint-free opening turns (deduped by store and title-shape) and pages deeper into the pool on turns that disclose nothing new, both aimed at making an uninformative turn still useful, which lexical's equivalent feature is switched off for (measured to move zero sessions, since its own early-turn exposure gate already made it moot).
• Intent Override is the one scenario that asks the customer to retract something. A hard filter can't safely un-exclude what it already dropped, so a short, generic override value appended to the AND-chain can silently carve the true target out of the pool before ranking ever sees it. Lexical's override instead erases and rewrites only the one attribute slot the new value belongs to, leaving every other piece of accumulated evidence untouched, a rewrite of belief, not a new hard constraint, so nothing the customer says can ever remove the right answer from consideration.

Both tracks share the ideas that took the score from the starter's 0.107 to ~0.96: constraint accumulation (every turn is evidence, never a rewrite), a coarse-category bucket pre-filter mirroring the evaluator's own bucketing logic, a linear reranker where disclosed evidence always dominates tiebreakers, and an exposure gate that shows only the single best candidate on the first turn or two, since a miss just costs a turn, but a wrong-rank hit is scored at that rank forever.

The whole thing is deterministic and dependency-free: standard-library Python only, sqlite3's FTS5 extension for the BM25 index, no network calls, no model downloads, no random draws.

Challenges we ran into

• Diminishing, information-free ties. In the sessions we still lose rank on, the customer's one disclosed phrase (e.g. "Buckle closure", "100% Cotton") appears identically in 6–300+ same-bucket products and there's genuinely no signal left in the session to break the tie. We priced this ceiling directly: even an oracle reranker over the existing candidate slate is only worth +0.022, so we stopped chasing it.
• Killing our own darlings. We built and fully measured a local-LLM reranking layer and a dense bi-encoder retrieval path. Both looked promising on paper and both were net negative in practice (the LLM rerank cost efficiency; the dense route recovered zero missed targets and only demoted correct ones), so we cut them rather than keep them for appearances.
• Overfitting to a 200-sample set. The lexical track's constants were swept against the public set and dropped ~0.018 composite on held-out data. We caught and reverted one clearly-overfit weight before submitting.
• Template-locked opener parsing. Both the routing decision and the bucket pre-filter key off the simulator's exact phrasing, so a differently worded private evaluator could misroute a session, mitigated with safe fallbacks, but not fully solved.

Accomplishments that we're proud of

• 1.000 HitRate@10 on public, 0.9988 on 800 held-out sessions, the target is essentially always in the returned list, on data the system was never tuned against.
• A composite score of ~0.96–0.97, roughly a 9x improvement over the starter baseline, with zero model calls and zero tokens, the entire agent runs in milliseconds per turn on one CPU core.
• A router we can actually justify. Every routing decision is backed by standalone, held-out numbers for both tracks, not intuition, including the one non-obvious case (Intent Override) where the "smarter" elimination approach loses to the simpler lexical rewrite.
• The discipline to say no. We fully built and measured an LLM reranker and a dense retrieval path, watched them lose, and cut them, instead of shipping complexity that looked impressive but didn't earn its place.
• Bit-for-bit reproducibility. No randomness, no network dependency, no live credentials — anyone can rerun our exact numbers on a bare Python interpreter.

What we learned

• Read the simulator before reaching for a model. The verbatim-quoting behavior of the customer simulator made this fundamentally a string-match problem, not a semantic one, and every dense/LLM approach we tried later confirmed that the hard way.
• Measure the split, don't assume it. We only trust the two-track router because we have standalone numbers for both tracks on data neither was tuned against. Without that, "obviously route by scenario" would have been a guess dressed up as an architecture.
• A held-out set changes what "improvement" means. One of our reranker weights looked like a real gain on the 200-sample public set and turned out to be pure overfitting once checked against the 800-session dev set.

What's next for The Elimination Engine

• A paraphrase-robust opener classifier. Right now scenario routing and bucket resolution key off the simulator's exact phrasing. A fuzzy classifier plus a bucket-resolution ladder with measured recall floors would make the agent far more resilient to a reworded private evaluator.
• A gated dense-retrieval fallback. Purely lexical matching only works because the simulator quotes verbatim. We'd add a dense retrieval path that fires only when lexical evidence is thin, validated so it can never demote an exact match, insurance against a less literal customer simulator.
• A discriminating signal outside the disclosed text. For the information-free tie groups, we'd explore catalog structure or co-listing patterns that survive conditioning on category and popularity, since every catalog-wide prior we tried so far (ratings velocity, popularity) evaporated inside a tie group.
• One shared index instead of two. The two tracks currently each build their own full-catalog FTS5 index, doubling cold start and memory. Merging them into a single shared index and matcher would cut operational cost with no accuracy loss.
• Move all tuning onto the held-out set. We'd retire the public 200-set as a tuning target entirely and treat it purely as a final check, to stop measuring against the same split we're selecting against.

Built With

• python
• sqlite
• sqlite-fts5