Second Thought

Second Thought is a shopping copilot that remembers what matters, forgets outdated preferences when you change your mind, and adapts its search as your wants changes

Comment

Inspiration

Shopping rarely happens in one perfect search. People change their budget, priorities, style, or even what they are looking for entirely.

That creates an important challenge for a conversational shopping agent: what should it remember, and what should it forget?

Second Thought was built around that idea. Instead of treating the whole conversation history as permanently valid, it keeps a compact, changing view of what the shopper currently wants.

How Second Thought Addresses the Problem

Second Thought is a conversational shopping copilot that adapts as the user reveals more information or changes their mind.

It handles the challenge in five main ways:

  • Preference tracking: it keeps a structured record of active search terms and explicit product requirements
  • Controlled forgetting: it removes outdated preferences when the shopper changes direction
  • High-information clarification: it first asks for the one or two requirements that matter most, so useful constraints can be revealed earlier
  • Adaptive retrieval: it searches a smaller candidate pool when intent is still vague, then expands the search once more constraints are known
  • Constraint-aware reranking: it gives more weight to rare, specific requirements that are more useful for identifying the right product

For example, if a shopper first asks for something formal but later says: “Actually, forget formal. I want something casual” Second Thought can remove the outdated preference rather than continuing to search using both “formal” and “casual”.

The core idea is that: Preferences are mutable state, not permanent conversation history

How I Built It

Second Thought uses a lightweight multi-stage recommendation pipeline.

1. Preference Ledger

Each conversation maintains a small internal state containing:

  • active search terms,
  • explicit requirements,
  • revoked preferences,
  • attributes already asked about, and
  • attributes the user has said they do not care about

This gives the agent a cleaner picture of the shopper's current intent without repeatedly relying on the entire raw conversation.

2. Controlled Forgetting

The agent detects common signs of an intent change, such as: actually, instead, forget, ignore, and changed my mind

When this happens, outdated preferences are removed from the active state.

3. High-Information Clarification

Instead of immediately asking many narrow questions, the agent first asks the shopper for the one or two requirements that matter most.

This was one of the strongest improvements during development because it reveals more useful information in fewer turns.

4. Adaptive BM25 Retrieval

The product catalog is indexed using SQLite FTS5, with BM25 used for first-stage retrieval.

The number of candidates depends on how much information is available:

  • fewer known constraints → smaller candidate pool for precision,
  • more known constraints → larger candidate pool for better recall

5. Rarity-Aware Reranking

Retrieved products are reranked using the user's explicit requirements.

Rare constraints are treated as more informative than common ones. For example, a requirement that matches only a few products provides more useful evidence than one that appears across most of the candidate set.

The rarity weight is:

w(c) = 1 + log((N + 1) / (df(c) + 1))

The final ranking combines the original BM25 position with the weighted constraint matches:

Score(p) = 1 / BM25Rank(p) + λ × Σ[Match(p,c) × w(c)]

Development Tools

I developed and tested Second Thought using:

  • Visual Studio Code for development,
  • Jupyter Notebook for catalog exploration and failure analysis,
  • GitHub for version control and the final public repository,
  • GitHub Desktop for managing commits and branches, and
  • Command Prompt / Python virtual environments for local execution and evaluation

APIs

The final submitted system does not use any external API.

It does not require:

  • OpenAI or another paid LLM API,
  • an external search API,
  • an external vector database, or
  • any hosted inference service

This keeps the final agent lightweight and reproducible

Libraries and Frameworks

The final agent mainly uses Python standard-library components, including:

  • sqlite3 for the in-memory FTS5 product index,
  • json for loading product metadata,
  • re for text and intent-pattern matching,
  • math for rarity weighting, and
  • pathlib for file handling.

SQLite FTS5 provides full-text search, while BM25 is used to rank first-stage product candidates.

During development, I also experimented with sentence-transformers for dense semantic retrieval, but this performed worse on the public evaluator and was not included in the final system.

Datasets and Assets

The project uses the product catalog and public evaluation sessions provided for the TikTok TechJam shopping-agent challenge.

The catalog contains approximately 50,000 products from the Amazon Clothing_Shoes_and_Jewelry category, including metadata such as:

  • product titles,
  • categories,
  • features,
  • details,
  • store information, and
  • descriptions

The public evaluator contains 200 shopping sessions across browsing, buying, intent-override, and boundary scenarios.

No additional labelled dataset was created for the final system.

Challenges

One of the hardest parts was deciding what the system should forget.

Remembering preferences is easy, but users often change their mind indirectly. I had to make controlled forgetting responsive enough to handle intent changes without deleting useful information too aggressively.

Another challenge was balancing recall and precision. Increasing the number of retrieved products helped recover harder targets, but it also introduced more noise. Adaptive candidate depth gave a better balance.

I also found that more complex approaches were not always better. Dense semantic retrieval sounded promising, but it often returned products that were conceptually related while missing exact attributes that mattered to the shopper.

That led me to keep the final architecture lightweight and evidence-driven.

What I Learned

The biggest lesson was that conversational shopping is not only a retrieval problem. It is also a state-management problem.

Strong recommendations depend on knowing:

  • which preferences are still valid,
  • which ones have been replaced,
  • which requirements matter most,
  • when to ask another question, and
  • how broadly to search at each stage

I also learned the value of ablation testing. Several more complex ideas reduced performance, while relatively simple improvements such as high-information clarification and explicit constraint reranking produced much larger gains.

Results

On the 200-session public evaluation set, Second Thought achieved:

Metric Result
Hit Rate @ 10 96.5%
MRR 0.6655
MTTC 2.73
Efficiency 0.8270
Recommended Technical Score 0.8476

Compared with the official starter baseline:

Metric Starter Second Thought
Hit Rate @ 10 12.5% 96.5%
MRR 0.0680 0.6655
MTTC 9.81 2.73
Technical Score 0.1067 0.8476

What I Would Improve Next

Given more time, I would focus on improving flexibility without losing the precision of the current system.

The main next steps would be:

  • gated semantic retrieval, where semantic candidates are only introduced when lexical search is uncertain,
  • more general natural-language preference extraction,
  • confidence-aware question selection,
  • richer preference state that tracks recency and preference strength, and
  • evaluation on more natural conversations with real shoppers

The goal would be to make Second Thought more flexible while keeping the simple, interpretable behaviour that worked well in the challenge.

Share this project:

Updates

Submission history