CartChatter

Conversation Example
Evaluator Score

Inspiration

Real shopping conversations aren't clean. A shopper might start vague ("I'm looking for a jacket"), reveal constraints gradually — material, then color, then budget — and sometimes contradict themselves entirely ("actually, forget cotton, I want polyester"). TikTok TechJam's Track 4 challenge asks for an agent that holds up under exactly this kind of messy, evolving conversation: searching a frozen 50,000-product catalog and finding a customer's hidden target product within at most 10 turns, with no UI, purely through an automated back-and-forth. We wanted to build something that treats a shopping conversation like a real conversation, not just a search bar with extra steps.

What it does

CartChatter holds a multi-turn conversation with a shopper. Each turn, it:

Parses whatever the customer just said, extracting any product attributes mentioned (material, color, budget, style, use case, size) Searches the catalog using everything learned about them so far, not just the latest message Decides whether to ask a clarifying question about something it doesn't know yet, or present its best current recommendations Returns up to 10 ranked products

How we built it

The pipeline: customer message > constraint extraction > dual-state conversational memory > SQLite FTS5 lexical retrieval (widened candidate pool) > a three-stage reranker (active-term coverage > per-slot coverage > exact phrase coherence) > top-10 ranked recommendations, with a proactive clarification question when useful.

No LLM or paid API is used anywhere in the pipeline — retrieval and ranking are both deterministic, running entirely on Python's standard library plus SQLite's built-in full-text search. This was a deliberate choice: zero marginal cost per query, no external dependency, and fully reproducible by anyone with just Python installed.

Our core architectural insight came out of a real bug. Early on, we used a single memory structure for two different jobs: "what does the customer currently want" and "what search evidence has this conversation produced." Those aren't the same thing. Consider: "I wanted cotton earlier, but actually I want polyester." The agent needs to recognize the current preference is polyester — but aggressively deleting every prior mention of "cotton" from the search itself throws away useful signal (in this dataset, both the old and new stated preferences are often genuine facts about the true target product). We solved this by splitting session memory into two concepts:

active_slots — the customer's current, live intent, used to decide what to ask next and to rerank results toward what they're asking for right now slots — all accumulated lexical evidence from the whole conversation, used only for the underlying search, and deliberately not erased when a preference changes

This distinction became the central idea of the system.

Challenges we ran into

We treated every proposed improvement as a hypothesis, not a fact — testing each one against the real 200-session evaluation set before keeping it. Several ideas that sounded correct on paper didn't survive contact with real data, and we reverted them:

Erasing old preferences on override. Textbook-correct dialogue design, but it measurably hurt ranking quality, for the reason above — old and new preferences are often both genuine facts about the target. A hard AND-filtered "strict" search track, meant to mirror a classic buying/browsing split. On the real 50,000-product catalog it surfaced loosely-related products ahead of better matches, dropping hit rate substantially. Stopping clarification questions once the candidate pool looked small. This increased average turns to conversion, because a small-but-wrong candidate pool gave the agent no way to recover.

Iterating this way — build, measure against real data, keep or revert — took longer than just shipping the "obvious" version of each idea, but it's directly why our final numbers hold up: Hit Rate@10 went from 12.5% (organizer's weak reference agent) to 88.0%, with MRR improving from 0.068 to 0.568 and average turns-to-conversion dropping from 9.81 to 5.50.

One remaining honest challenge: our final phrase-coherence reranking stage improved match quality at the cost of higher evaluation runtime (roughly 53 seconds → 85 seconds per 200-session run). We suspect an additional lookup against an unindexed field is the cause, though we haven't profiler-confirmed the exact bottleneck yet — it's the next thing on our list.

Accomplishments that we're proud of

Our biggest accomplishment is turning a deliberately simple, fully deterministic system into something that performs surprisingly well on messy, multi-turn shopping conversations. We improved Hit Rate@10 from 12.5% to 88.0%, MRR from 0.068 to 0.568, and reduced average turns-to-conversion from 9.81 to 5.50.

We're especially proud of the architectural insight that came from debugging our conversational memory. Separating active intent from accumulated search evidence allowed CartChatter to handle contradictions like “actually, forget cotton, I want polyester” without throwing away potentially useful evidence from earlier turns.

We're also proud that the entire pipeline runs without an LLM or paid API. Retrieval, memory, clarification, and ranking are deterministic and reproducible using Python and SQLite FTS5. This keeps the system inexpensive, transparent, and easy to run and evaluate.

Finally, we're proud of the way we iterated: instead of assuming an approach was correct because it sounded theoretically right, we repeatedly tested our ideas against the real evaluation set. Several seemingly sensible improvements actually hurt performance, and we were willing to revert them.

What we learned

The biggest lesson was that real conversational intent is not the same as the latest user message. A shopper's preferences can evolve, contradict each other, or reveal information gradually, so a useful agent needs to distinguish between what the shopper currently wants and everything they've told us that could still be evidence about the target product.

We also learned that more restrictive search isn't necessarily better search. Our strict AND-filtered retrieval approach seemed intuitive, but on the real catalog it removed useful candidates and actually reduced hit rate.

Another important lesson was that clarification should be driven by uncertainty, not just candidate count. A small candidate pool can still be completely wrong, so stopping questions simply because retrieval returned fewer products made the agent less able to recover.

Most importantly, we learned to treat every optimization as a hypothesis to validate, rather than an assumption. Measuring each change on the actual 200-session evaluation set helped us discover what genuinely improved the system and what only looked good on paper.

What's next for CartChatter

Our immediate priority is performance optimization. The phrase-coherence reranking stage improved recommendation quality, but increased evaluation runtime from roughly 53 seconds to 85 seconds per 200-session run. We'd like to profile this properly, identify the unindexed lookup we suspect is responsible, and make the reranking pipeline faster without sacrificing accuracy.

Beyond that, we'd like to make CartChatter's conversational strategy more adaptive — particularly improving when and what it asks, so that clarification questions provide the maximum information with as few turns as possible.

We'd also like to explore whether the same architecture can generalize beyond this fixed catalog and evaluation setup. The goal is to keep CartChatter deterministic, explainable, and inexpensive, while making its retrieval and conversational reasoning robust enough for larger catalogs and more diverse shopping scenarios.