Fable7

Inspiration

TechJam’s conversational search track challenges teams to solve a problem that looks like conventional retrieval but is fundamentally about managing uncertainty. We received a frozen catalog of 50,000 clothing products and a simulated customer who knows exactly what they want but reveals it one turn at a time. Our agent gets ten turns to ask clarifying questions and place the hidden target product in its top-10 list. This is very similar to how most of us shop and how we ask one question after another to narrow down exactly what we need.

The obvious interpretation was to follow well-established systems and embed everything and try to improve ranking. We initially followed that path and even fine-tuned a TinyBERT cross-encoder. However, our ablation studies consistently showed that retrieval was not the main bottleneck. Once a few attributes had been disclosed, the target was almost always already in our candidate pool. The real errors came from ranking too confidently while dozens of products remained literally indistinguishable based on what the customer had revealed.

This inspired us to build a conversational search system that knows when it does not know.

What it does

Our agent finds the target in 100% of the 200 public sessions, always at rank 1, in approximately two turns on average. It achieves a TechnicalScore of 0.978, compared to 0.107 for the official BM25 starter.

The system accumulates customer constraints across turns, retrieves and reranks relevant products, and asks further questions while the available evidence still matches multiple possibilities. Once the evidence uniquely identifies a product (or the conversation reaches its final turn) it releases the full ranked list.

The submitted agent runs entirely offline. It makes no LLM calls, requires no network connection or GPU, consumes zero tokens, and completes the entire 200-session evaluation in approximately 45 seconds on a laptop CPU. Its inference cost is $0.00.

How we built it

We designed the final system as a funnel with five main components:

Stateful dialogue tracking. Constraints accumulate across turns. An intent override such as “actually, forget that...” wipes the stale opening request while retaining later disclosures.
Category-scoped lexical retrieval. Three BM25 routes: conjunctive, phrase, and disjunctive search a SQLite FTS5 index. Their results are combined using reciprocal-rank fusion (RRF) and constrained to the exact coarse category parsed from the opening message.
An exact-evidence lane. A given constraint was that customers quote catalog attributes verbatim, so we created an inverted index over normalized feature and detail values. This acts as a high-precision channel alongside BM25.
A learned reranker. A 16-feature pairwise logistic model orders an 80-candidate pool. It was trained using scikit-learn, shipped as a plain JSON weight file, and executed using only the Python standard library.
Abstention. For every product, we precompute the ordered “evidence card” that a customer could disclose about it. While the observed dialogue prefix still matches multiple products, the agent returns a single best candidate and asks another question instead of padding its response with ten guesses. When the evidence becomes unique (or the agent reaches the final turn) it releases the full list. This decision alone increased MRR from 0.85 to 1.0.

We developed the system using:

Cursor and VS Code on macOS and Windows
AI coding agents, including Claude and Codex, for implementation and experiment execution, with every change gated by the official evaluator score and a unit test suite before merging
Git and GitHub, with one branch per experiment checkpoint
Python virtual environments and unittest for a 23-test regression suite

The submitted runtime uses only the Python 3.10+ standard library, including sqlite3 with FTS5 for BM25 retrieval. It has no third-party runtime dependencies and makes no external API calls—no LLM APIs, vector databases, or hosted services. We evaluated the LLM-agent route against the public baselines, but it performed worse on both score and cost, so we removed it.

For offline training and experimentation, we used:

scikit-learn and NumPy for linear pairwise rerankers with five-fold cross-validation
PyTorch and Hugging Face Transformers for fine-tuning the cross-encoder
ONNX and ONNX Runtime for exporting and quantizing the model into a 4.49 MB QUInt8 asset, with a parity audit across the PyTorch, ONNX, and quantized outputs
safetensors for checkpoint storage

Our datasets and assets included:

Amazon Reviews 2023 from the McAuley Lab at UCSD, specifically Clothing_Shoes_and_Jewelry: the organizer-provided frozen catalog of 50,000 products and 200 labeled public sessions, sampled from the official 5-core leave-last-out split
Self-generated synthetic sessions: more than 5,000 deterministic, seeded sessions whose targets exclude all 200 public targets, together with a 500-session hard-case set and a paraphrase harness for wording-robustness checks; these required no manual labelling or scraped data
cross-encoder/ms-marco-TinyBERT-L2-v2 from Hugging Face: the pretrained base model that we fine-tuned using catalog-derived pairs, with its pinned revision and SHA-256 manifests committed for reproducibility

Challenges we ran into

Our biggest challenge was realizing that stronger semantic retrieval and more sophisticated neural ranking did not necessarily produce a better conversational agent. The target product was often already present in the candidate pool, but the agent lost points by making premature ranking decisions when the disclosed evidence could not yet distinguish among several products.

We also had to manage changing intent across multiple dialogue turns. A customer might retract the original request without invalidating everything they said afterward, so the system needed to remove stale opening constraints while preserving later disclosures.

Category parsing introduced another edge case. Restricting retrieval to an exact coarse category improves precision when the category is clear, but could eliminate the correct answer when parsing does not apply. We addressed this by failing open to global search.

Finally, one of our most time-consuming approaches was the fine-tuned neural reranker. This was ultimately outperformed by deterministic dictionary lookup. Instead of keeping it merely because of the effort invested, we shipped it but disabled it by default.

Accomplishments that we're proud of

We are particularly proud of 3 main 'breakthroughs' we had during the development phase:

Firstly, our idea to create an inverted prefix index to be able to determine a set of products based on a tuple of narrowing keywords allowed us to make significant improvements in the reranking phase while only having minor preprocessing cost.

Next, our abstention strategy was especially impactful. By returning one best candidate while the evidence remained ambiguous instead of padding the ranking with unsupported guesses until the number of results reached 10, we increased MRR from ~0.85 to ~0.95.

Lastly, when evidence was ambiguous, instead of returning the same item again and again, by rotating out previously seen items, we were then able to improve our MRR further from ~0.95 to 1.0.

We are also very proud that our agent achieved a 0.978 TechnicalScore and found the correct target in all 200 public sessions, always at rank 1 and in approximately two turns on average. This significantly outperformed the official BM25 starter score of 0.107.

We are furthermore proud that the system delivers this performance entirely offline, with no APIs, network services, GPU, third-party runtime packages, token usage, or inference cost. The complete public evaluation runs in approximately 45 seconds on a laptop CPU.

We kept our development process rigorous with a target-blind 150/50 split of the public set, running holdout evaluations only after freezing each configuration. We also evaluated against thousands of generated validation sessions whose targets were disjoint from every public label. Every experiment including failed approaches is recorded in EXPERIMENTS.md in the repository.

The project is backed by a 23-test regression suite, one Git branch per experiment checkpoint, reproducible model revisions and SHA-256 manifests, and a parity audit across the PyTorch, ONNX, and quantized model outputs.

What we learned

We learned that conversational product search is not simply a retrieval problem. A system must distinguish between failing to find the target and lacking enough evidence to rank it confidently. In our case, uncertainty management mattered more than increasingly complex retrieval models.

We also learned to trust carefully designed ablations and holdout measurements over our attachment to a particular technique. The neural reranker required substantial effort, but deterministic dictionary lookup performed better. Believing our own measurements meant disabling the more sophisticated model.

Finally, we learned that efficiency and accuracy do not have to be opposing goals. A stateful, deterministic system built around lexical retrieval, exact evidence, lightweight learned ranking, and calibrated abstention can outperform a more expensive LLM-based approach while remaining fast, reproducible, and completely offline.

What's next for Fable7

Fable7’s next step is to strengthen the system beyond the current frozen evaluation setting. We plan to expand the paraphrase harness and hard-case suite to test a wider range of customer wording, incomplete descriptions, corrections, and ambiguous attribute combinations.

We also want to evaluate the architecture on larger and more varied product catalogs beyond Clothing_Shoes_and_Jewelry, where category boundaries and product attributes may be less structured. This will test whether our evidence-card and abstention approach generalizes to new domains.

Finally, we plan to continue investigating lightweight learned components that preserve the current system’s zero-API, standard-library runtime. The shipped but disabled cross-encoder provides a reproducible foundation for future experiments, but any new component will only be enabled if frozen holdout measurements show that it improves accuracy, ranking quality, turn efficiency, and runtime cost.