CyborgBoiS: Context-Aware Shopping Copilot

Inspiration

Traditional e-commerce search works best when shoppers already know the right keywords. Real customers rarely start that way. Someone may begin with “I need shoes for a trip,” later reveal that they expect long walks, then add a budget or replace an earlier preference entirely. A useful shopping assistant must understand this evolving intent instead of treating every message as an isolated query.

The TechJam challenge makes that problem measurable. An agent must navigate a 50,000-product catalog, support Buying, Browsing, Intent Override, and Boundary conversations, and place the hidden purchased product in the first ten results within at most ten turns. Success depends on recall, ranking precision, and conversational efficiency—not merely generating fluent text.

What it does

CyborgBoiS is a stateful shopping copilot that converts a multi-turn conversation into an active search plan. It remembers valid requirements, separates hard constraints from soft preferences, removes superseded context after an intent change, and preserves explicit exclusions and budgets.

On every turn, the agent can both ask one useful clarification and return its current best ten recommendations. This is important because clarification and retrieval are complementary: shoppers should receive useful options while the agent continues reducing uncertainty.

The system works fully offline through a deterministic fallback. When model credentials are available, an optional LLM planner distils the active context into validated JSON, and an optional LLM reranker improves the short fused candidate list. Network or model failures never prevent the offline pipeline from responding.

How we built it

1. Stateful conversational memory

Each session stores the conversation history, category, material, color, size, style, brand, use case, features, budget, exclusions, declined attributes, and anonymized profile preferences. Answers containing multiple requirements are kept as separate facts instead of being flattened into a lossy summary.

Intent overrides selectively remove superseded soft preferences while preserving independently stated hard constraints and the product category. This prevents both stale-context retrieval and over-aggressive memory resets.

2. Intent routing and search planning

The agent identifies whether the customer is Buying with explicit constraints or Browsing more openly. A deterministic planner always produces a semantic query, hard constraints, soft preferences, exclusions, and a clarification decision.

If Groq or Anthropic is configured, the conversation and active state are sent to an LLM that returns the same strict JSON schema. All fields are validated, and malformed or unavailable model responses fall back automatically.

3. Multi-route retrieval

The frozen catalog is indexed in memory through SQLite FTS5 and an optional local BGE embedding matrix. The agent retrieves a recall-oriented candidate pool from several complementary routes:

exact phrases for highly specific product details;
grouped precision queries that combine category and active constraints;
recent-constraint precision queries;
balanced category-plus-preference BM25 retrieval;
broad lexical retrieval for recall; and
dense semantic retrieval using BAAI/bge-small-en-v1.5.

All candidate identifiers originate from the frozen catalog.

4. Constraint-aware fusion and reranking

Candidates are deduplicated and reranked using route position, exact phrase evidence, category match, hard-constraint coverage, soft-preference coverage, budget compliance, exclusion penalties, and small rating/profile signals.

Dense retrieval is deliberately recall-only. Cosine similarity can introduce a product into the candidate union, but it cannot displace a product with stronger exact catalog evidence merely because the score scales differ.

When enabled, the LLM reranks only the short fused list and is restricted to the supplied catalog identifiers.

5. High-value clarification

The question policy is not tied to public labels or a fixed evaluator-specific order. It examines the live candidate pool and estimates how much each attribute would divide the remaining products. The highest-information relevant attribute that has not already been answered or declined is selected.

The agent still returns its current Top 10 while asking that question, reducing the mean number of turns needed to find the target.

Challenges we ran into

Remembering meaning without retaining stale intent

Keeping every message verbatim improves recall, but it can also preserve a preference the customer explicitly replaced. We solved this with structured facts, provenance by turn, and selective override handling rather than a single ever-growing prompt.

Combining lexical and dense scores

BM25 and cosine similarity have unrelated scales. Directly adding them caused semantic candidates to displace exact matches. We therefore use route-based candidate generation and catalog-evidence reranking, treating dense retrieval as a recall source rather than unquestioned ranking authority.

Asking fewer, better questions

A fixed questionnaire wastes turns and can overfit a simulator. The final policy estimates attribute information gain from the current candidates and adapts the next question to the active search space.

Remaining reliable without network access

External LLMs improve semantic planning but introduce latency, cost, and availability risk. Every LLM operation is optional and validated, with a deterministic offline fallback covering the complete Agent contract.

Accomplishments that we're proud of

Increased public Hit Rate@10 from 0.125 to 0.840 without runtime access to public labels or ground truth.
Reduced MTTC from 9.81 turns to 3.855 turns.
Built a complete offline path that requires neither an API key nor a vector database server.
Loaded and queried a local 50,000-product BGE index through a normalized in-memory matrix.
Supported abrupt intent changes, negative preferences, multi-value answers, and boundary “no preference” responses.
Added automated tests for memory, overrides, exclusions, LLM-plan validation, dense candidate fusion, embedding metadata, and response-contract behavior.

What we learned

Lexical and dense retrieval don't combine by just adding scores

We initially normalized and summed BM25 and cosine-similarity scores into one ranking signal. It looked reasonable on paper and broke in practice: dense matches on loosely related products would outscore exact catalog matches because the two scales don't mean the same thing. We learned to stop treating dense retrieval as a ranking authority and instead use it purely as a recall source — it's allowed to add a candidate to the pool, but only lexical and catalog evidence can decide where that candidate lands. Route-based candidate generation followed by evidence-based fusion turned out to be far more robust than any single blended score.

The best question isn't the next question on a checklist

We started with a fixed clarification order (category, then size, then budget, ...). It felt thorough but wasted turns — asking about budget when the candidate pool already agreed on price, for instance. Estimating how much each unanswered attribute would actually divide the current candidate set, and asking that one, cut turns substantially without any change to retrieval quality. The lesson: a clarification policy should be a function of the live state, not a static script, even when the static script feels "complete."

Offline-first forced better design, not just a fallback

Making the deterministic path fully capable — not a degraded stand-in for an LLM-backed path — was more work upfront but paid off everywhere. Because the agent had to work with zero network calls, every optional LLM step (planning, reranking) had to justify itself as a strict improvement over a already-correct baseline, and its output had to be validated against the same schema either way. We learned that designing the "no API key" path first, rather than last, produces a system that's easier to reason about and test, and it made failures in the optional LLM path harmless rather than catastrophic.

What's next for CyborgBoiS: Context-Aware Shopping Copilot

Train or calibrate a compact offline cross-encoder on permitted development examples for stronger final reranking.
Learn question value from candidate entropy, expected constraint coverage, and observed turn cost rather than relying on domain priors.
Add field-specific or multi-vector embeddings for titles, categories, features, and descriptions to reduce long-document dilution.
Quantize the embedding model and cache repeated state encodings to reduce CPU latency and memory.
Calibrate confidence and stopping thresholds through cross-validation and broader adversarial paraphrase tests.
Produce transparent recommendation explanations showing which active facts each product satisfies.