BayesPilot - Deterministic Bayesian Probabilistic Agent

Introduction
Benchmark & Results
High-level Architecture
Detailed Architecture
Level 1 - Category Posterior
Level 2 - Bounded Item Likelihood Fusion
Optimal K - Expected Utility formula U(k)
Hyperparameter Tuning on TPE
Hyperparameter Tuning on TPE
Pruned Components
Demo Video
Thank You

Problem Statement

Track 4 asks us to build a conversational shopping agent that can identify a customer's hidden target product from a large e-commerce catalog within a limited number of dialogue turns.

At first glance, this may look like a retrieval problem: given a customer message, search the catalog and return the most relevant products.

We interpret it differently.

The real challenge is to continuously maintain and update a belief about what the customer wants, while deciding when there is enough information to make a recommendation.

What makes the problem difficult

Unclear intent: Early messages are often broad, incomplete, or underspecified. The system cannot assume that the first query contains enough information to identify the correct product.
Changing preferences: New information may add, refine, contradict, or completely replace an earlier preference. A conversational recommender therefore needs to treat dialogue history as evolving evidence rather than a fixed collection of keywords.
Vocabulary mismatch: Customers naturally paraphrase, use slang, abbreviations, and informal descriptions, while the catalog stores relatively fixed product titles and attribute values. Exact keyword matching alone is therefore not sufficient.
Catalog uncertainty: Several product categories may appear equally plausible from the first message. Eliminating candidates too aggressively can permanently remove the correct product before later turns provide the evidence needed to distinguish it.
Timing pressure: Recommending too early with insufficient evidence can hurt ranking quality and MRR, while asking too many questions increases MTTC and reduces efficiency. The system must continuously decide whether another conversational turn is more valuable than recommending now.

From this perspective, the task becomes a sequential probabilistic decision problem:

Maintain a belief over possible products.
Update that belief whenever new customer evidence arrives.
Preserve uncertainty instead of eliminating plausible candidates too early.
Decide at every turn whether to ask for more information or return recommendations.

Traditional keyword search approaches struggle because they largely treat the conversation as an unweighted bag of words. At the other extreme, modern LLM-based agents can provide flexible language understanding, but introduce additional latency, cost, non-determinism, and the possibility of losing or hallucinating constraints across turns.

This led us to a different engineering question:

Can we solve the conversational recommendation problem using a mathematically principled, deterministic probabilistic system — preserving uncertainty, accumulating evidence over time, and explicitly deciding when to recommend — without relying on a heavy runtime model?

That question led to BayesPilot.

What it does

BayesPilot is a deterministic, offline probabilistic shopping agent that finds a single hidden target product within a 50,000-item Amazon catalog in 10 conversational turns or fewer.

Instead of prompting an LLM, BayesPilot executes two-level Bayesian inference:

Calibrated Category Belief: Evaluates the posterior distribution across 1,115 catalog categories, expanding the candidate pool until 85% of posterior probability mass is covered—safely shrinking 50,000 products down to ~250 candidates in under 1 millisecond.
Bounded Multi-Source Evidence Fusion: Fuses exact catalog metadata, normalized attribute-value pairs, lexical overlap, and soft-card Jaccard similarity into an item log-posterior with temporal age decay (0.9 per turn) and a 0.02 likelihood floor.
Decision-Theoretic Recommendation Depth Policy: Dynamically calculates the optimal recommendation depth (k* between 0 and 10) on every turn by maximizing an Expected Utility function that balances immediate reciprocal rank against the continuation value of asking another question.

Verified Benchmark Performance

BayesPilot maintains strong retrieval quality across all three evaluation settings while keeping end-to-end conversational inference in the single-digit to low-millisecond range.

Evaluation Set	Sessions	Hit@10	MRR	MTTC	TechnicalScore	Latency / Session
Official Public Set	200	1.0000	0.9942	2.19	0.9744	16.9 ms
Generated Template Set	2,800	0.9911	0.9783	2.64	0.9562	7.8 ms
Free-Form Set	800	0.9912	0.9801	2.62	0.9572	12.7 ms

Runtime Efficiency

Across the evaluated datasets, BayesPilot achieves:

7.8 ms ≤ Latency per Session ≤ 16.9 ms

with:

0 LLM API calls
0 prompt tokens
0 completion tokens
$0.00 API cost
No GPU required

This demonstrates that high-quality multi-turn product retrieval can be achieved with a lightweight deterministic probabilistic system without sacrificing inference speed.

How we built it

BayesPilot is architected as an in-memory, zero-dependency four-tier probabilistic pipeline:

[ Customer Message ]
        │
        ▼
┌─────────────────────────────────────────────────────────────┐
│ 1. Deterministic NLP & State Tracker                        │
│    Regex Templates (0ms) ➔ Fuzzy Ontology Normalizer        │
│    Session State Tracker (Slot decay γ=0.9, Demote 0.35)    │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Level 1: Bayesian Category Belief (1,115 Categories)     │
│    IDF Overlap Scoring ➔ Softmax (T=2.0) ➔ 85% Prefix Mass  │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Level 2: Bounded Item Likelihood Fusion                  │
│    Exact + Soft-Card + Lexical ➔ Floor L_min=0.02 ➔ Mask -∞ │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────────────────────┐
│ 4. Decision-Theoretic Recommendation Depth Policy           │
│    k* = argmax U(k) [Immediate MRR vs Continuation Value V] │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
[ Ranked ASINs (0..10) + Optimal Clarifying Question ]

1. Level 1: Category Posterior Belief

The customer's opener message x is first scored across all 1,115 taxonomy categories:

$$s_c(x)=W_c(x)\cdot coverage_c(x)+3.0\cdot I_c(x)\cdot W_c(x)$$

Here:

W_c(x) is the shared-token IDF mass.
coverage_c(x) is the proportion of category tokens matched.
I_c(x) is 1 when category c is explicitly quoted by the customer, and 0 otherwise.

Raw category scores are converted into calibrated probabilities via Softmax with temperature T = 2.0 and a catalog-share prior π_c:

$$P(c\mid x)=softmax(s_c(x)/2.0+0.25\log\pi_c)$$

Rather than making a brittle top-1 guess, we accumulate categories until we cover 85% of the total posterior mass (τ = 0.85):

$$\sum_{c\in C_\tau}P(c\mid x)\ge 0.85$$

2. Level 2: Bounded Multi-Turn Evidence Accumulation

To prevent missing catalog fields from multiplying probabilities to zero, evidence likelihoods are lower-bounded at L_min = 0.02:

$$\log L_r(i\mid e)=\log(\max(0.02,\exp(g_r(s_r(i,e)-1))))$$

Exact card matches receive gain g_exact = 3.2, while soft-card Jaccard receives gain g_soft = 1.5 with threshold J_soft ≥ 0.34. Lexical tokens provide the baseline overlap signal.

Across dialogue history D_t, evidence is accumulated in log-space with temporal age decay:

$$\log P_t(i\mid D_t)\propto\sum_{e\in D_t}0.9^{t-turn(e)}(\log L_{main}(i\mid e)+\log L_{soft}(i\mid e))$$

If an item was recommended in turn t−1, inspected by the customer, and the dialogue continued, that item is proven wrong and masked to negative infinity:

$$\log P_t(i)\leftarrow-\infty$$

3. Decision-Theoretic Recommendation Depth Policy

At each turn, the agent dynamically selects the depth k* between 0 and 10 that maximizes Expected Utility:

$$U(k)=\sum_{j=1}^{k}p_j/j+(1-\sum_{j=1}^{k}p_j)V$$

The first term represents the expected reciprocal-rank reward of recommending immediately. The second represents the value of continuing the conversation when the correct product is not yet inside the returned list.

The optimal recommendation depth is:

$$k^*=argmax_{0\le k\le 10}U(k)$$

The continuation value V models expected future information gain:

$$V=\max(0,0.75h-0.0667)$$

where:

$$h=d^s$$

The decay factor d depends on how confidently the deterministic parser understands the conversation:

d = 0.8 when templates match
d = 0.2 when the parser is blind

The variable s represents consecutive stalled turns where no new constraint was extracted.

Early turns (High V): The agent returns fewer items (k* = 0 or 1) and asks questions because waiting for more evidence has greater expected value.
Late turns or stalled info (Low V): V falls toward zero, causing the agent to automatically widen its recommendation list up to the full top-10 (k* = 10).

4. Hyperparameter Tuning & Scientific Discipline

We exposed 8 core system constants to a Tree-structured Parzen Estimator (TPE) offline.

TPE selects the next candidate configuration by maximizing the ratio between the density of promising trials and the density of the remaining trials:

$$x^*=argmax_x\;l(x)/g(x)$$

where:

$$l(x)=p(x\mid G)$$

$$g(x)=p(x\mid R)$$

Here:

G represents configurations belonging to the good-performing trials.
R represents the remaining trials.

To eliminate search noise and prevent overfitting, we enforced a 95% paired-bootstrap confidence interval gate.

A proposed configuration is adopted only when:

$$CI_{95\%}^{low}(\Delta S_j)>0$$

where ΔS_j is the change in TechnicalScore produced by proposal j.

In other words, a parameter change is accepted only when the lower bound of its 95% confidence interval is above zero.

When recent proposals failed to clear this statistical gate, we rejected them and kept our incumbent baseline.

Spec-Driven + Test-Driven Development

Every major BayesPilot component followed the same engineering loop:

Specify → Define the test → Implement → Benchmark → Ablate → Keep or Remove

We combined Spec-Driven Development (SDD) with Test-Driven Development (TDD) so that implementation was guided by both the competition contract and measurable system behaviour.

Spec-Driven Development (SDD)

Before implementing a component, we first defined what it was supposed to do:

What input does it receive?
What output should it produce?
What state can it modify?
How should it behave when information is missing, contradictory, or ambiguous?
How must it conform to the official evaluator and API contract?

The competition specification was our source of truth. A system that scores well but breaks the required interface or dialogue behaviour is not a valid solution.

Defining these contracts first also made parallel development easier: separate team members, worktrees, and AI-assisted development sessions could work on different components without constantly renegotiating interfaces.

Test-Driven Development (TDD)

For a recommendation system, our most meaningful tests were not only unit tests — they were measurable changes in end-to-end system performance.

Before adopting a new idea, we first defined what it was expected to improve:

Hit@10, MRR, MTTC, TechnicalScore, robustness, or latency.

Every proposed change then had to face:

the official evaluator, unmodified;
an ASIN-disjoint held-out split, preventing gains from product memorisation;
free-form, paraphrased, misspelt, and edge-case inputs;
component-level ablation tests;
runtime and latency measurements;
and a 95% paired-bootstrap confidence gate for parameter changes.

A feature was not accepted because it looked sophisticated or produced one promising run.

It stayed only if the measurements supported it.

This process gave us the confidence to reject our own ideas. The popularity prior contributed +0.000000, BM25 introduced noise, the trained reranker increased inference cost for almost no gain, and several LLM-assisted variants performed worse than the deterministic path.

None survived the tests, so none shipped.

That is why the final BayesPilot codebase is relatively small:

it contains the components that survived both the specification and the measurements.

Development Tools, APIs & Libraries

Development Tools

We developed BayesPilot primarily in VS Code, while using Codex and Claude Code across parallel development sessions to implement, debug, review, and cross-check different ideas.

When experiments were independent, we used Git worktrees to maintain isolated copies of the repository. This allowed multiple approaches to be developed and benchmarked in parallel without interfering with the main implementation.

Our development process followed a simple loop:

Build → Measure → Compare → Keep or Remove

A component was only kept if it demonstrated measurable improvements in retrieval accuracy, ranking quality, robustness, or latency.

Development Tool	Usage
VS Code	Primary development environment
Codex	Parallel implementation, debugging, experimentation, and code review
Claude Code	Parallel implementation, debugging, review, and cross-checking
Git / Git Worktrees	Isolated parallel development of competing approaches
GitHub	Version control and submission repository

API Usage

BayesPilot uses no external APIs at runtime.

The final submitted agent runs entirely offline with:

0 LLM API calls
0 search API calls
0 cloud endpoints
0 vector database calls
0 API keys
0 prompt or completion tokens
$0.00 inference API cost

During development, we initially experimented with a school-provided LLM API. API rate limits slowed down experimentation, so we later self-hosted a model locally using vLLM Metal on Apple Silicon.

These LLM experiments were used to evaluate alternative approaches and generate additional robustness test data. They are not part of the final BayesPilot inference pipeline.

Libraries & Frameworks

Library / Framework	Usage	Scope
Python 3.11.9	Core implementation, parsing, indexing, dialogue-state management, and evaluation logic	Runtime
NumPy 2.3.3	Numerical operations, probability calculations, and Bayesian scoring	Runtime
Optuna 4.9.0	Tree-structured Parzen Estimator (TPE) hyperparameter optimisation	Offline tuning
vLLM Metal	Local LLM serving on Apple Silicon during exploratory experiments	Experimentation

Final Runtime

The final BayesPilot runtime is intentionally lightweight:

Python 3.11.9 + NumPy 2.3.3

No LLM, GPU, vector database, learned reranker, or external API is required during inference.

During development, we also implemented and benchmarked several alternatives, including learned rerankers, dense semantic embeddings, BM25 retrieval, BLaIR/SVD representations, GBDT/LightGBM reranking, and popularity-based priors.

These components were deliberately removed after ablation showed that their additional complexity or latency did not provide sufficient improvement over the final bounded Bayesian fusion approach.

Dataset Used

The official benchmark contains only 200 public sessions, which is useful for validating the system but too small to give us confidence that BayesPilot generalises beyond the provided examples.

To test this more rigorously, we created two additional evaluation datasets from the provided 50,000-product catalog.

Provided Dataset

Dataset	Sessions	Purpose
Public Set	200	Official Track 4 benchmark used to evaluate Hit@10, MRR, MTTC, and TechnicalScore

Additional Evaluation Datasets Created by Us

Dataset	Sessions	Description
Generated Template Set	2,800	Catalog-grounded conversations generated using the structure of the official dialogue templates. The data is split 60/20/20 and kept ASIN-disjoint, ensuring that products seen during tuning do not appear in the held-out test split.
Free-Form Set	800	A harder robustness set containing natural, non-template customer language including paraphrases, slang, incomplete descriptions, unusual wording, and conversations that may begin with little or no useful product information.

Why we created them

A high score on only 200 structured sessions could potentially come from overfitting to the public conversation format.

We therefore wanted to answer two additional questions:

Does the recommendation algorithm generalise to unseen products?
Does it still work when customers stop speaking in fixed templates?

The Generated Template Set tests product-level generalisation across a much larger set of catalog items, while the Free-Form Set stress-tests BayesPilot against more realistic conversational language.

For the free-form dataset, we used LLM generation to create diverse customer utterances grounded in catalog products. The LLM is used only for offline dataset generation — it is not part of BayesPilot's inference pipeline.

Challenges we ran into

The Brittle Zero Problem in Bayesian Multiplications:
Early prototypes multiplied raw probabilities. If a product listing omitted a color tag that the user requested, its probability dropped to 0.0, permanently destroying target recall. We solved this by transitioning entirely to bounded log-likelihoods with an explicit floor (L_min = 0.02).
Taxonomy Hierarchy Ambiguity:
When a user requests "women's tops & tees", the catalog contains 7 sibling subcategories (Tunics, T-Shirts, Knits, etc.). An argmax filter guesses wrong ~85% of the time. We solved this with 85% cumulative mass prefix pruning (τ = 0.85), allowing the candidate pool to hold all sibling categories until subsequent turns narrow the field.
Resisting Optimizer Overfitting:
TPE frequently proposed micro-adjustments that produced +0.0003 gains on training data. Enforcing the 95% paired-bootstrap gate proved that many of these were random fluctuations. Rejecting them protected our 0.9562 held-out generalization score.
Pruning Redundant Systems:
We initially integrated BM25, a popularity prior, and dense vector embeddings (BLaIR/SVD). Through rigorous ablation, we discovered that once soft-card Bayesian evidence was active, the popularity prior contributed +0.000000 and BM25 added noise. Having the discipline to delete them kept the system lightweight and lightning fast.

Accomplishments that we're proud of

🥇 0.9744 TechnicalScore on the official public leaderboard with 1.0000 Hit@10 and 0.9942 MRR.
⚡ Sub-15ms Latency: Evaluating the entire 3,800-session test suite takes just 58 seconds total, compared to 30+ minutes for LLM-based agents.
💰 Zero Compute/API Cost: Runs 100% offline in pure Python without GPUs, cloud endpoints, or external vector databases.
🛡️ Mathematical Determinism & Zero Hallucination: Every recommendation decision is mathematically explainable, auditable, and repeatable.

What we learned

"Extracting a value is worthless unless a downstream decision consumes it": In early experiments, parsing complex route labels didn't improve accuracy because the underlying ranking likelihood didn't need them. Eliminating unused extraction reduced latency without score loss.
Principled Bayesian Fusion Beats Generic LLM Prompting: In structured e-commerce catalog discovery, a calibrated probabilistic model with bounded likelihoods and dynamic depth policies decisively outperforms generative models in accuracy, speed, and cost.
The Value of Rigorous Ablations: Building a component is only half the work; having the scientific discipline to measure, ablate, and prune components such as BM25 and popularity bias is what creates production-ready systems.

What's next for BayesPilot - Deterministic Offline Probabilistic Agent

Multi-Modal Image Likelihood Fusion: Extend Level 2 bounded evidence to incorporate precomputed visual feature vectors such as textures, patterns, and silhouettes into log L_visual without introducing heavy runtime models.
Dynamic Client-Side Personalization: Allow the prior term log π_c to dynamically adapt to on-device user browsing history with zero privacy leakage.
WebAssembly / Edge Deployment: Compile the pure Python deterministic engine to WebAssembly (Wasm) and C++ for sub-millisecond, offline execution directly inside mobile e-commerce apps and browser tabs.

Live Demo

Try BayesPilot directly through our interactive demo:

🔗 Live Demo: https://ewencheung.github.io/BayesPilot/

The demo provides a simple front-end for exploring BayesPilot's conversational recommendation flow and seeing how the deterministic inference pipeline narrows down products across multiple turns.