GhostLab

Runtime Architecture: How each shopper message is processed through state updates, adaptive routing, retrieval, ranking, and guidance.
GhostLab Offline Optimizer: Pipeline configurations compete, improve, and survive progressively tougher evaluation rounds.
In GhostLab racing, different complete pipeline configurations compete against each other on the same development sessions.
200 official sessions: GhostLab improves substantially over the organizer’s BM25 starter on the official public evaluation.
- 550 untouched sessions: GhostLab maintains its improvement on unseen held-out sessions, supporting generalization beyond development data.

Project summary

GhostLab is a local conversational shopping agent for targeted buying and open-ended discovery. Each turn, it updates shopper intent; retrieves candidates through keyword, category, and semantic search; removes confirmed constraint violations; and ranks the merged pool with learned and selective local-LLM scoring. It returns up to ten validated recommendations, switching to bounded retrieval plus one clarification question for overly broad requests.

GhostLab searches the provided 50,000-product Amazon catalogue using a 2,200-session corpus: 200 official sessions, 1,000 public-scenario variations with new targets, and 1,000 independently generated scenarios with new profiles and targets. Lineage-safe grouping reserves 1,650 sessions for development and 550 sessions as a one-time final selection set.

Behind the shopping agent is the GhostLab offline optimization engine. It preserves the required adaptive workflow, then searches compatible additions and parameter settings across retrieval, fusion, ranking, profile use, and dialog guidance. Progressive F0/F1/F2 racing spends small evaluation budgets on broad exploration, promotes only evidence-backed survivors, and freezes the strongest architecture-valid configurations for final comparison.

How GhostLab addresses the problem statement

I. Core Architecture: Intent Routing and Hybrid Pipeline

1A — Dual-Track Intent Routing

GhostLab makes an observable route decision after every message: Buying emphasizes precise evidence, while Browsing enables broader semantic discovery. A deterministic router keeps each decision reproducible and inspectable through a confidence score and reason.

Signals: Constraint strength, exclusions, specificity, corrections, exploratory language, and browsing keywords help resolve mixed-intent requests.
Fallback: Uncertain decisions use the BM25-led Buying route to limit semantic drift.

1B — Multi-Route Retrieval → Semantic Ranking

Both routes combine BM25 (exact wording), category matching (catalogue structure), and E5 (semantic meaning). Buying emphasizes explicit requirements, while Browsing emphasizes scenarios that may not match catalogue wording.

Multi-view Browsing: Multiple E5 views represent the complete request, use case, and product or style hypotheses, reducing dependence on one interpretation.
Evidence and constraints: Duplicate products are merged while retaining each source’s scores and ranks. Confirmed contradictions are removed, while candidates with unknown metadata remain eligible.
Ranking: A route-aware union GBDT ranks the merged pool. The champion uses RRF fusion, Browsing-only SmolLM2-1.7B semantic ranking over the Top 10, and a hash-verified Top-10 residual reranker.

II. Dialog Strategy: Multi-Turn Scenario Evolution

2A — Dynamic State Machine

GhostLab maintains one structured conversation state, implemented as State V2. Compatible requirements accumulate, corrections replace only the affected requirement, and a new shopping goal clears incompatible information from the previous intent.

Tracked context: State V2 records raw history, category, positive and negative constraints, no-preference attributes, evidence sources, and shown products.
Intent override: Each new shopping goal starts a new intent epoch, separating its constraints and recommendation history from earlier goals.
Consistent processing: After every update, the state is frozen into an immutable turn context shared by all downstream stages.

2B — Proactive Guidance

Before full semantic expansion, GhostLab runs a cheap keyword and category preview to detect overly general requests. When overloaded, it switches to a bounded path that still returns preliminary recommendations alongside one useful clarification question.

Overload detection: Few resolved constraints, weak score separation, high candidate uncertainty, and broad category or facet coverage indicate overload.
Cutoff: The bounded path uses reduced retrieval, constraint validation, and a lightweight safe ranker, skipping the normal union GBDT and local LLM.
Clarification: GhostLab selects one unresolved supported attribute and avoids repeating answered, declined, or no-preference questions.

III. Self-Evolution: Dynamic Context Programming

3A — Runtime Adaptation

After every message, GhostLab distils the current conversation state and supplied user profile into one structured turn context. Conversation state drives retrieval, routing, ranking, and clarification; profile preferences receive a small, conflict-safe ranking influence when the request is ambiguous.

Conflict handling: Explicit requirements from the current conversation take priority, while profile preferences help only when the current request is ambiguous.
Profile updates: GhostLab emits confidence- and source-aware updates that can be persisted across sessions in a real-world deployment; isolated evaluation does not exercise this capability.
Pending optimization: ⚠ Profile influence, profile-aware query views, and clarification-question suppression.

3B — Adaptive Orchestration

A fixed orchestrator rebuilds the shopping workflow from the latest turn context after every message. It adapts which predefined components are used and how strongly they contribute, while preserving the required processing order.

Runtime choices: The orchestrator selects Buying or Browsing, normal or bounded retrieval, route-specific retrieval weights and budgets, local-LLM activation, and whether to include a clarification question.
Reliable completion: Component failures trigger bounded fallbacks, and final validation runs before recommendations and selected actions are committed to the conversation state.

Offline Optimization Engine: How GhostLab Evolves the Agent

GhostLab separates the system into a fixed adaptive architecture and an optimizable implementation layer. Required capabilities—State V2, Buying/Browsing routing, BM25/category/E5 retrieval, constraint filtering, source-aware merging, local-LLM capability, proactive guidance, runtime adaptation, fallback, and atomic commit—cannot be removed or reordered. The engine improves the agent by adding compatible techniques, replacing only equivalent implementations, and tuning bounded parameters inside that workflow.

Technique registry: The current catalogue records 88 techniques. Nineteen are compulsory parts of the adaptive workflow, while 20 are safe promotable options. The remaining entries are retained as controls, research procedures, or unavailable experiments rather than being presented as deployable improvements.
Optional additions: Promotable techniques include RRF, weighted and rank-stack fusion; quality and profile priors; catalogue pseudo-relevance feedback; facet diversity; MiniLM retrieval support; dense-view selection; catalogue normalization; and bounded residual or auxiliary rankers. Compatibility rules prevent an optional technique from bypassing constraints, replacing the compulsory source-aware union ranker, or changing the required workflow order.
Combination search: The engine first evaluates the fixed control, each safe add-one candidate, and compatible pairs. Strong survivors can expand into higher-order combinations such as a proven base plus RRF and a residual reranker. A translated warm start may seed known-good settings, but it is executed through the current architecture rather than reusing an obsolete runtime.
Conditional tuning: Hyperparameters are tuned only when their parent technique is active. Examples include retrieval depth and source weights, RRF constant, diversity strength, profile influence, local-LLM depth and weight, residual-reranker depth, and clarification thresholds. This avoids spending trials on parameters that cannot affect a candidate.

Progressive F0/F1/F2 Racing

Phase	Sessions per candidate	Purpose
F0	330	Broad, inexpensive screening of the control, additions, ablations, and compatible combinations
F1	825	Re-evaluation of surviving structures with stronger paired evidence and conditional local tuning
F2	1,650	Full-development confirmation and finalist eligibility checks

Every phase compares candidates on the same ordered session subset using paired rewards. Promotion considers TechnicalScore, Hit Rate@10, MRR, scenario and source slices, constraint safety, fallback behavior, and latency. Clearly dominated candidates are pruned early; sparse evidence is held for more data rather than treated as permanent failure. One to three eligible F2 challengers may be frozen, with three acting as an upper bound rather than a requirement.

The selected GhostLab Champion keeps the complete adaptive architecture and combines the learned union GBDT with RRF fusion, bounded SmolLM2 semantic ranking, and a hash-bound Top-10 residual reranker. Once frozen, the same configuration and assets are used unchanged for the one-time 550-session final selection; holdout results are never fed back into optimization.

IV. Evaluation Matrix: Product and Efficiency Metrics

Each session is evaluated against its known purchased product using the organizer’s unchanged evaluator. Following the challenge framing, we report Coverage through Hit Rate@10, Precision through MRR, and Efficiency through MTTC, together with the combined TechnicalScore.

Official public-set comparison

The organizer’s BM25 starter and the frozen GhostLab configuration are evaluated on the 200 official public sessions. This provides a direct comparison with the organizer’s published benchmark.

Metric	Organizer BM25 starter	Fixed Adaptive Architecture	GhostLab Champion
Hit Rate@10	0.125	0.97	0.975
MRR	0.068034	0.572325	0.688401
MTTC	9.81	2.775	2.765
Normalized Efficiency	0.119	0.8225	0.8235
TechnicalScore	0.106710	0.821197	0.858720

One-time final selection

Of the 2,200 sessions, 1,650 are used for training and optimization, while 550 are reserved for final evaluation. Related synthetic sessions remain in the same partition to prevent data leakage. After GhostLab is frozen, it and the organizer starter are evaluated once on the same 550 sessions.

Metric	Organizer BM25 starter	Fixed Adaptive Architecture	GhostLab Champion
Hit Rate@10	0.190909	0.961818	0.965455
MRR	0.101387	0.568150	0.676361
MTTC	9.110909	2.723636	2.689091
Normalized Efficiency	0.188909	0.827636	0.831091
TechnicalScore	0.163652	0.816881	0.851854

Final-selection safeguards

Protected validation: Holdout results are checked overall and across Buying, Browsing, Intent Override, Boundary, and dataset-source groups.
Promotion: Against the Fixed Adaptive Architecture control, the GhostLab Champion improved TechnicalScore from 0.816881 to 0.851854 (+0.034973; paired 95% CI: +0.024089 to +0.044697), with zero output constraint violations and zero fallbacks.
Evaluation integrity: No models, thresholds, or workflow settings are changed after the holdout results are revealed.

Key Innovations

GhostLab’s innovation is controlled adaptation at two levels: conversation context reshapes the runtime strategy each turn, while offline experimentation selects the strongest validated configuration.

Evidence-Preserving Ranking: BM25, category, and multi-view E5 evidence survives merging, enabling route-specific ranking across Buying precision and Browsing discovery.
Productive Overload Cutoff: A cheap preview detects overly broad requests before full semantic retrieval. GhostLab switches to bounded retrieval and safe ranking, returning preliminary recommendations with one high-value clarification question.
Whole-Pipeline Optimization: GhostLab trains rankers and compares complete combinations of routing, retrieval, diversity, ranking, LLM, and clarification settings using grouped folds. Weak configurations are eliminated early, while finalists must pass quality, safety, and latency gates.
Offline-First Execution: After setup, retrieval, ranking, and model inference run locally and in memory, avoiding runtime API costs, external vector databases, and network dependencies.
Two-Level Generalization Corpus: Two synthetic datasets separately test new products under familiar public scenarios and new products under unfamiliar shopper templates, using agent-independent target selection and lineage-grouped splits to prevent leakage.

Development tools used

The project was developed using:

Python 3.10–3.13 (3.12 recommended)
Git and GitHub for version control
uv for reproducible dependency management
Command-line development across Linux, macOS, and Windows through WSL2
pytest for automated testing
Ruff for linting and formatting checks
mypy for static type checking

APIs and external services used

GhostLab uses no external inference APIs at runtime. Retrieval, ranking, and language-model inference run locally and in memory.

Hugging Face Hub is used only during setup to download version-pinned model assets. Their hashes are verified before the models are loaded in offline mode.

Libraries and frameworks used

NumPy: Feature, embedding, and ranking operations
Pydantic: Configuration, contract, trace, and evidence validation
scikit-learn: GBDT training and experimental learned rankers
PyTorch: Local neural-model and LLM inference
Hugging Face Transformers: Loading and running local language models
Sentence Transformers: E5 embeddings and cross-encoder scoring
SQLite FTS5: In-memory, field-weighted BM25 retrieval

Datasets and assets used

Datasets

GhostLab uses the organizer’s frozen 50,000-product catalogue and builds a 2,200-session corpus from the official development set and two complementary synthetic datasets. All sessions follow the official JSONL schema and are replayed through the same counterfactual simulator.

Official dataset: [200 sessions] 80 Buying, 80 Browsing, 30 Intent Override, and 10 Boundary sessions.
Public-scenario variations: [1000 sessions] Five new-product variations of each official scenario, preserving its shopper context while assigning unique catalogue targets. This tests new products under familiar shopping patterns.
Independent-template dataset: [1000 sessions] Five sessions from each of 200 newly generated shopper and information-disclosure templates. This tests unfamiliar profiles and target products.

Dataset Construction and Splitting

Leakage control: Synthetic targets are selected without running or scoring GhostLab.
Lineage-safe split: Development uses 150 official, 750 public-scenario, & 750 independent-template sessions (1650 total); the one-time final selection set uses 50, 250, & 250 respectively (550 total). Related families remain together within the same partition and nested fold.
Scenario balance: Each synthetic dataset contains 400 Buying, 400 Browsing, 150 Intent Override, and 50 Boundary sessions.

Local Model Assets

intfloat/e5-small-v2: Dense semantic retrieval
HuggingFaceTB/SmolLM2-1.7B-Instruct: Selected Browsing-only semantic ranker. Qwen models are historical experiments.
MiniLM cross-encoder: Bounded fallback semantic scoring
GBDT assets: Union candidate ranker and Top-10 residual reranker

Model revisions, paths, licences, and SHA-256 hashes are recorded for reproducibility.

Conclusion

GhostLab turns conversational shopping into a controlled, measurable path from uncertain intent to confident product choice. Its evidence-led design supports rigorous evaluation and practical deployment, providing a reproducible foundation for trustworthy conversational commerce.