TokenMaxx Copilot

Inspiration

Most product search fails the moment a shopper doesn't know the right keyword.

The shopper describe an occasion: "something for a beach wedding", Or change their mind halfway: "actually, not gold, I want silver"

And keyword search has no answer.

Therefore, we (jollibee888) wanted to build an agent that handles this the way a good sales associate would: tracking what you said, forgetting what you retracted, and asking the one question most likely to help find what you're actually looking for.

The deeper motivation was measurement honesty. The competition's scoring formula is:
$$\text{TechnicalScore} = 0.50 \times \text{HitRate@10} + 0.30 \times \text{MRR} + 0.20 \times \text{Efficiency}$$

Where Efficiency rewards finding the target in fewer turns.

We noticed early that the public evaluator generates customer messages that contain phrases lifted almost verbatim from the target product's own catalog description.

This means that a well-tuned lexical retriever can score very high without truly understanding natural language.

But we decided to build a system that wins both the benchmark, and a separate 250-session honest set where customers describe needs in their own words, not catalog terms, to measure what the pipeline actually generalises to.

What it does

TokenMaxx Copilot searches a 50,000-product clothing, shoes, and jewelry catalog over a conversation of up to ten turns. On each turn it can ask one structured follow-up question, return ranked product recommendations, or do both.

Remembers when you change your mind — maintains an explicit constraint ledger that retires old preferences on intent override, rather than blindly accumulating the full transcript
Asks smarter questions over time — a GuidanceLearner measures how much each clarification question actually narrowed the search, and reweights future question priorities accordingly
Works without any API or cloud dependency — the full scored path runs on SQLite and the Python standard library, $0 operating cost
Diversifies results for fairer coverage — an MMR diversifier surfaces distinct styles instead of ten near-identical items from the same brand (but is off by default for testing on this frozen catalogue)

Representative verified flow:

Turn 1 shopper: I'm looking for Jewelry Necklaces. A key requirement is: Material:alloy.
Turn 1 agent:   asks for one more differentiating preference, reveals one candidate.
Turn 2 shopper: reveals a distinctive product feature.
Turn 2 agent:   returns the correct product at rank 1. Model tokens used: 0.

How we built it

The architecture is a staged pipeline with clean separation between components:

message + profile
  → intent and constraint-state update
  → BM25 retrieval (+ optional dense/expansion routes)
  → need-satisfaction ranking (with cross-encoder)
  → belief, clarification, and reveal policy
  → contract-shaped response

Retrieval uses in-memory SQLite FTS5 BM25 as the always-on baseline, with an optional BGE dense embedding track fused via reciprocal-rank fusion. The two tracks are weighted differently for buying versus browsing intent.

State is maintained as an explicit constraint ledger — not a raw transcript. Every preference is stored with its polarity, turn, and weight. On intent override, old constraints are soft-demoted rather than evicted, because the evaluator constructs messages partly from the target's soft preferences — keeping a partial signal helps.

Ranking scores candidates by how well their catalog text covers the active constraint phrases. A local cross-encoder (ms-marco-MiniLM-L-6-v2) runs by default and is gated by a regime router: on turns where exact phrase matches are strong, the lexical coverage signal takes precedence; on turns where it is weak, the cross-encoder provides high-precision reranking. This prevents the cross-encoder from diluting the verbatim signal when it is already decisive.

Self-evolution is implemented via the GuidanceLearner: it measures the entropy drop in the belief model after each clarification question and reweights future question priorities with an exponential moving average. Dormant in offline evaluation (each session resets), active in a live deployment.

Result diversity is handled by an MMR diversifier that protects the confident top picks and fills remaining slots by penalising title-token similarity to already-selected items. Off by default for the competition benchmark, on for a real storefront.

Challenges we ran into

Separating benchmark performance from real-language performance. The verbatim overlap between simulator phrasing and product descriptions inflates lexical retrieval scores in a way that would not hold for real shoppers. We built the honest set specifically to audit this gap — and used it to decide which components to ship on the scored path versus flag as production-only.

The adaptive reveal problem. Revealing results too early locks the target's MRR at whatever rank it holds on turn 1. Revealing too late costs Efficiency. We implemented a confidence-gated reveal policy that holds back a short list while belief is low, and reveals unconditionally from turn 4 onward. This alone moved MRR from 0.705 to 0.861.

Intent override without reintroducing retired terms. When a shopper says "forget the red, I want blue," naively appending to the constraint ledger leaves "red" in the retrieval query. We handle this with soft demotion — old phrases stay at reduced weight because they are still true of the target's soft preferences, just no longer the primary signal.

Keeping optional components from making startup fragile. Dense embeddings, the cross-encoder, and Gemini each sit behind a separate availability check. If any one fails to load, the system falls through to the next layer without raising. The verified public run used zero model tokens.

Accomplishments that we're proud of

Public evaluation: Hit Rate@10 0.965, MRR 0.852, MTTC 2.905, TechnicalScore 0.9001 — zero model tokens, no API key
Honest-set diagnostic (self made 250 sessions in realistic "customers' own words", not catalog terms): Hit Rate@10 0.908, MRR 0.664, TechnicalScore 0.8071
A GuidanceLearner that learns which clarification questions narrow the search fastest — implemented and running, not a roadmap item
An MMR diversifier for fairer result coverage — implemented, tested, and deliberately left off for the competition with a documented reason
138 passing automated tests, clean lint, and a fully reproducible evaluation path

What we learned

Measure what you're actually optimising. Early on we noticed the public evaluator generates customer messages using phrases lifted directly from the target product's catalog description. A retriever that memorises popular phrasing can score 0.96 Hit@10 without understanding a single natural sentence. That realisation changed how we made every subsequent decision — we stopped asking "does this raise the score?" and started asking "does this help because the pipeline is better?"

Good engineering and good benchmarking are not the same thing. The MMR diversifier, the GuidanceLearner, the correction-aware ledger — none of these move the public leaderboard number meaningfully, and some hurt it. But they are the features that would matter in a real deployment. Separating "scored path" from "production path" explicitly, and documenting the reason for each flag, was one of the more useful structural decisions we made.

Ablation discipline is worth the time. We ran a measured ablation before shipping every feature to the scored path. Several things we were confident would help turned out to be neutral or negative (semantic coverage hurt robustness; APR hurt disambiguation). Without the numbers we would have shipped them and not known why the score moved. The flag ledger in src/agent.py is the artefact of that discipline — every flag has a measured justification or an explicit "unproven" label.

The private set is the real test. Everything we built for the honest set — the cross-encoder, regime routing, structured constraints — was built knowing the private evaluation sessions would not have the same verbatim overlap. Public score is a guardrail. The honest set is the actual signal.

What's next for TokenMaxx Copilot

The architecture is ready to move beyond a benchmark. The immediate next steps for a real deployment:

Turn on the production features. The MMR diversifier and GuidanceLearner are implemented and tested but disabled for scoring. A live deployment would enable both and instrument them — measuring whether diverse result lists reduce pogo-sticking, and whether the question reweighting actually improves conversion over the first few thousand sessions.

Replace the simulated evaluator with real users. The honest set is a proxy, not a substitute. The next real test is a small-scale pilot with actual shoppers — collecting implicit feedback (clicks, add-to-cart, session abandonment) to validate whether MTTC in evaluation maps to conversion rate in practice.

Solve the cold-start problem properly. Right now, a new user gets the same question priority order as everyone else until the GuidanceLearner accumulates enough signal. For a real deployment, bootstrapping from category-level purchase data or onboarding questions would meaningfully reduce first-session friction.

Multi-catalog and multi-category support. The current pipeline is tuned for one frozen catalog. Extending to a live, updating inventory — with new products, price changes, and retired SKUs — requires an incremental indexing strategy and drift monitoring to catch when the BM25 or embedding index goes stale.

Privacy and session handling at scale. The durable profile store is currently a local JSON file with a 45-day decay. Production needs a proper key-value store, a defined data-retention policy, and user-facing controls to view or delete their preference history — especially relevant in markets with strong data protection expectations.