TTSC | Devpost

Inspiration

Online shopping search often fails when users cannot describe everything they want in one query. We wanted to build a shopping assistant that learns through conversation—asking useful follow-up questions, remembering preferences, and refining results without relying on expensive external APIs.

What it does

TTSC is an offline conversational product-search agent. It:

Maintains structured preferences across multiple turns
Handles browsing, direct buying requests, changing intentions, and “no preference” answers
Combines BM25 keyword search with local semantic embeddings
Reranks products using the user’s accumulated constraints
Avoids repeatedly showing the same unsuitable products
Runs without an LLM, network connection, API key, or token cost

How we built it

We built a deterministic retrieval pipeline around four main components:

Conversation state: Each message updates an immutable representation of the user’s category, requirements, exclusions, unanswered attributes, and intent version.
Hybrid retrieval: SQLite FTS5 BM25 finds strong lexical matches, while a local INT8 BGE-small ONNX model retrieves semantically related products. Their results are combined using weighted reciprocal-rank fusion.
Evidence-based reranking: Retrieved products are compared with the user’s explicit constraints, including category, material, color, features, exclusions, and budget. Exact and consistent evidence is prioritized while uncertain cases fail open to the original retrieval order.
Clarification and presentation: The agent requests additional information when useful and tracks previously displayed products so later turns can explore new candidates.

The evaluation score is:

Technical Score = 0.50 × Hit Rate@10 + 0.30 × MRR + 0.20 × Efficiency

Our latest public evaluation achieved:

Hit Rate@10: 1.000
MRR: 0.996
MTTC: 2.350 turns
Technical Score: 0.972
Reported token usage: 0

Challenges we faced

The hardest problem was balancing recall, ranking quality, and conversation length. Returning many products improves the chance of including the target, but can place it too low. Returning too few can improve rank while accidentally removing the correct product.

We also had to handle incomplete catalog evidence. A missing attribute does not necessarily mean a product violates the user’s requirement, so aggressive hard filtering performed poorly. We instead developed tiered evidence ranking with conservative fallbacks.

Another challenge was preventing overfitting to the public evaluator. Several changes improved isolated examples but failed on target-disjoint validation sets. We kept experiments on separate branches and rejected changes that did not generalize.

What we learned

We learned that stronger models are not automatically the biggest source of improvement. Reliable state management, candidate recall, deterministic evidence handling, and careful evaluation were often more valuable than adding model complexity.

We also learned that conversational search is a decision problem as much as a retrieval problem. The system must decide when to search, when to ask, what evidence to trust, and how much of the ranked list to reveal.

Most importantly, every ranking improvement needs to be tested against Hit Rate and conversation efficiency. Optimizing one metric in isolation can easily reduce the overall quality of the system.