TikTalk

Agent Pipeline

Inspiration

We've all had that shopping moment: you know roughly what you want, but not what to type into the search bar. Maybe it's "something comfortable for a long flight" or "a bag that looks professional but isn't too formal." Traditional e-commerce search expects the perfect keywords upfront, while real shoppers discover what they want along the way, asking questions, narrowing options, and sometimes changing their minds completely.

When we saw TikTok TechJam's Track 4 challenge, we wanted to bridge that gap between a search box and a salesperson who actually listens. But instead of throwing a heavyweight model at every turn, we asked a different question: how much intelligence can come from better retrieval itself?

So we set ourselves a constraint: no language model in the core decision loop. The result is a conversational shopping agent that is deterministic, offline-capable, lightweight, and fully traceable.

What it does

TikTalk is a conversational shopping agent that turns product search into an adaptive, multi-turn dialogue.

At every turn, TikTalk returns a ranked Top-10 from a 50,000-item catalogue while deciding whether to recommend directly or ask for more information.

It combines BM25-style lexical retrieval, structured attribute filtering, and conversation-state tracking to continuously update what the shopper wants. When users change their minds, TikTalk can replace outdated constraints rather than allowing old preferences to keep influencing results.

Instead of simply classifying shoppers as "buying" or "browsing," TikTalk uses an intent-certainty signal. When intent is clear, it ranks the strongest candidates directly; when uncertainty remains, it diversifies the Top-10 across unresolved attributes and asks clarifying questions.

Every recommendation is grounded in catalogue evidence, making the process both relevant and explainable.

How we built it

We began by analyzing the baseline agent and evaluator to understand the Agent API, scoring methodology, and key performance gaps.

We framed recommendation as a belief that updates every turn:

Pₜ(p) = P(target = p | evidence so far, history, catalogue)

We then built a lightweight retrieval pipeline using SQLite FTS5, indexing product titles, features, descriptions, categories, and details. The system combines BM25 retrieval, typed conversational evidence, structured filtering, candidate-state tracking, override detection, and clarification logic.

Our development process was highly iterative:

v1 — TS 0.820: introduced cumulative conversation state and open-first clarification.
v2 → v2.8 — TS 0.682 → 0.855: added two-route BM25 retrieval, typed evidence, risk-gated scoring, override handling, and diversified Top-10 coverage.
v2.8.1 → v2.8.4 — TS ≈ 0.86: refined question selection through information-gain scoring and depth-gated clarification.
v2.8.4.1 — Final: expanded override detection for more natural "changed my mind" turns.

We also ran ablation tests and removed components that did not measurably improve performance.

Final Public Evaluation

Metric	Result
Technical Score	0.866
HitRate@10	1.000
MRR	0.666
Mean Turns to Conversion	2.70
LLM Tokens Used	0

Because 800 evaluation sessions were private, we created our own held-out datasets: 300 holdout_mirror sessions and 200 holdout_natural_cards sessions. Changes were accepted only when they showed positive paired improvement with a 95% confidence interval and no regression across scenario types.

Challenges we ran into

One of the hardest problems was handling changing intent across multiple turns. A message might introduce a new constraint, remove an old one, or completely reverse a previous preference. We therefore built explicit conversation-state and override logic so TikTalk could update what matters now instead of simply accumulating keywords.

Another challenge was balancing retrieval accuracy with generalization. Small changes in BM25 field weights could significantly affect rankings, while only 200 evaluation sessions were public. To avoid overfitting, we tested changes against our synthetic holdout sets and relied on ablation and statistical evidence rather than individual examples.

We also discovered that historical preferences were less useful than expected. Using profile history produced no measurable ranking improvement because the benchmark profiles were too abstract and often unrelated to the shopper's current mission. Instead of forcing personalization into the system, we kept history as a bounded supporting signal that never overrides current intent.

These challenges shaped a central principle behind TikTalk: current intent comes first, every recommendation must be grounded in catalogue evidence, and every added component must justify itself through measurable improvement.

Accomplishments that we're proud of

We're proud that TikTalk achieved a 0.866 Technical Score and 1.000 HitRate@10 while using zero LLM tokens. Every recommendation remains traceable to concrete retrieval signals, catalogue evidence, and filtering decisions rather than opaque model reasoning.

We're also proud of the evaluation infrastructure we built beyond the provided public sessions. Our synthetic holdouts and statistical testing allowed us to distinguish real improvements from changes that simply looked promising.

Most importantly, our team combined technical and non-technical backgrounds effectively. By dividing responsibilities around individual strengths and continuously sharing findings, we were able to iterate quickly under a tight hackathon deadline.

What we learned

TikTalk taught us that more complexity does not automatically mean better intelligence. Several ideas that looked promising produced little measurable improvement, and ablation testing helped us identify what actually mattered.

We also learned the importance of evidence-based engineering. Building our own holdout sets and validating changes statistically gave us a much stronger basis for decision-making than intuition alone.

At the team level, clear communication and effective division of responsibilities allowed us to iterate quickly while keeping the system coherent.

Most importantly, conversational search does not necessarily require generating more language. Better state management, better retrieval signals, and better questions can already make search substantially more intelligent.

What's next for TikTalk

Our first priority is structural override detection. We want to replace phrase-based detection with a between-turn contradiction signal so TikTalk can recognize changing preferences more robustly.

Next is ranking precision. Although the target appears in the Top-10 on every public session, it still falls below rank 3 roughly one-third of the time. An optional semantic reranking layer could improve MRR while keeping our deterministic retrieval system as an offline fallback.

We also want to validate at larger scale with more diverse synthetic sessions, add proactive clarification when the candidate pool remains too large, and revisit real personalization once richer item-level purchase histories become available.

Ultimately, we see TikTalk evolving into a lightweight conversational commerce system that understands what shoppers want now, adapts when that changes, and can explain exactly why it recommended what it did.