Performance
Playground
Problem Formulation
Case Study

ARC — Ask · Rank · Commit

An answerability-aware, evidence-grounded sequential decision agent for conversational shopping

ARC is an offline multi-turn shopping agent built for TikTok TechJam 2026 Track 4.

One hidden product. A 50,000-item catalog. At most ten turns. The shopper starts with incomplete requirements and may decline a question or change their mind.

ARC treats this as a sequential decision problem, not a one-shot search query. On every turn it makes three connected decisions:

ASK the question that is answerable and worth its extra turn;
RANK products using accumulated shopper evidence, not popularity alone;
COMMIT only as many results as the current evidence can safely support.

System	Hit Rate@10	MRR	MTTC	TechnicalScore
Organizer weak baseline	0.125000	0.068034	9.810	0.106710
ARC (submitted)	1.000000	1.000000	1.980	0.980400

These results use the unchanged organizer evaluator over all 200 public sessions. Every one of the 200 targets is returned, and every one of them is returned at rank 1. The submitted runtime uses zero model tokens, zero network calls, no GPU, and only the Python standard library.

Why shopping needs more than retrieval

A retrieval system can contain the right product and still create a poor shopping experience:

A vague request may leave hundreds of equally plausible products.
A high-discrimination question may be useless if the shopper cannot answer it.
Showing ten weak candidates too early can end the session with the target at a poor rank.
Repeating products that were already rejected wastes both turns and trust.
A changed preference can make otherwise correct conversation memory stale.

The challenge metric makes these failures concrete. Hit Rate rewards finding the target, MRR rewards putting it near the top, and MTTC charges for every additional turn. ARC therefore optimizes the interaction loop, not only a retrieval score.

One line of arithmetic sets the whole policy. Because a session is scored at the turn the target first appears, demoting a rank-1 result to rank 2 costs 0.30 × 0.5 / 200, while saving an entire turn earns only 0.20 × 0.1 / 200. One lost rank is worth 7.5 saved turns. Any design that shows more products sooner has to clear that bar, and almost nothing does.

Proving the turn budget is spent optimally

Claiming a policy is good is easy; we wanted to know whether it was finished. tools/turn_audit.py replays the organizer's own loop while capturing the agent's complete ranked list on every turn, then compares what ARC achieved against what a perfect output gate could have achieved.

The result is the strongest statement we can make about the decision layer:

Turns lost to the output gate: 0. Across all 200 sessions there is not one case where the target had already reached rank 1 and ARC withheld it. The target is exposed on exactly the turn it first becomes the best candidate.

The remaining MTTC is therefore bounded by disclosure, not by policy:

Structural bound	Cost
`intent_override` cannot score before its override turn	floor`3.600` turns, ARC achieves `3.667`
`boundary` spends one turn on the “I have no preference” reply	10 sessions, unavoidable
90 of 200 sessions open with zero constraints	turn 1 is a prior, not a ranking problem

We also checked, product attribute by product attribute, whether anything separates the target from the items ranked above it on turn 1 — review count, average rating, feature count, detail count, description length, price presence, title length, and first-availability date. All of them are coin flips. Turn 1 is an information limit, not a modelling failure.

This is why the first-hit distribution is as compressed as it is:

First hit on turn	1	2	3	4	5
Sessions	76	75	28	19	2

Why we deliberately did not put an LLM in the runtime

The challenge permits LLMs, but it does not require one. We chose the smallest system that directly addresses the measured bottleneck.

In this benchmark, shopper messages reveal catalog-derived constraints and a hit is an exact ASIN match. The dominant uncertainty is which useful constraint has not yet been revealed—not how to generate fluent prose.

What this task requires	Why an LLM is not the default solution here
Exact catalog-valid ASIN ranking	Fluent text does not guarantee exact identifier retrieval or ordering
Persistent constraints and overrides	These need explicit, testable state transitions rather than implicit prompt memory
Metric-aware question and output policy	An LLM does not automatically optimize Hit@10, MRR, and turn cost
Reproducible official scoring	Hosted inference adds network, credential, latency, cost, and nondeterminism risks
Grounded explanations	Generated rationale may sound plausible without matching the actual reranker

The intelligence is concentrated in deciding what evidence to acquire, how to update it, and when it is sufficient—not in generating more fluent prose.

This is not a claim that language models are never useful. It is an architectural boundary: use deterministic catalog evidence for the common path, and introduce a small local language or embedding fallback only when an ambiguous Top-N case shows a measured gain. Today the grounded parser achieves 1.00 Hit@10 and MRR on the natural-paraphrase robustness panel, while passing every audited wording variant. This is not an anti-LLM position: we would add a model when it earns its latency and complexity with a measured gain on unresolved language cases. Until then, putting one on the critical path would add operational risk without addressing the dominant failure mode.

How this maps to the Track 4 directions

The organizer's suggested techniques are possible tools, not a checklist. Our design uses the parts that improve the shopping decision and leaves speculative complexity disabled.

Track direction	ARC implementation	Shopper-facing outcome
Buying vs. Browsing routing	Scenario-aware shelf and question routing	Decisive buyers are served immediately; vague browsers are clarified
Hybrid retrieval and reranking	Shelf retrieval plus lexical, typed, signature, and bounded-prior evidence	Exact constraints beat generic popularity
Structured state and intent override	Confidence-weighted multi-turn memory with atomic history reset	Preferences accumulate without trapping the shopper in an old intent
Adaptive clarification	Answerability-aware metric value of information	The agent avoids questions that cost a turn but add no usable evidence
Failure detection and strategy switching	Proven-miss exclusion, refusal-aware pivot, late exact-tie rotation	Rejected products are not repeated and long-tail ties can recover
Indistinguishable-candidate planning	Finite-horizon batch-size optimization with final-turn Hit@10 insurance	Exact intent twins are tested at rank one instead of committed at a weak rank
Cold-start ranking	Review-count prior only before the first shopper constraint	A vague first turn is useful without letting popularity override intent
Low latency and token cost	Deterministic, offline, standard-library runtime	No API outage, credential, GPU, or per-query model cost
Transparent explanations	Evidence certificates and verified minimal counterfactuals	Engineers can inspect why the action and rank changed
Safe personalization	Aggregate profile support exists, but its ranking weight is disabled	Unproven profile correlations cannot override explicit intent

Dense semantic retrieval and profile weighting remain optional extensions. They were not enabled simply to match a suggested architecture; a new component must earn its complexity across public, popularity-matched, and uniform long-tail diagnostics.

Evaluation

Official public set

Scenario	n	Hit Rate@10	MRR	MTTC
Buying	80	1.000000	1.000000	1.487500
Browsing	80	1.000000	1.000000	1.775000
Intent override	30	1.000000	1.000000	3.666667
Boundary	10	1.000000	1.000000	2.500000
Overall	200	1.000000	1.000000	1.980000

The organizer weak baseline needs 9.81 turns on average; ARC needs 1.980, a reduction of 7.83 evaluator turns while raising MRR from 0.068034 to 1.000000.

Target-disjoint diagnostics

The public 200 are the only labels anyone can see, which makes them easy to overfit. We therefore hold ARC to two frozen panels whose targets never appear in the public set:

Diagnostic	n	Hit Rate@10	MRR	MTTC	TechnicalScore
Popularity-matched	800	1.000000	0.984496	2.12125	0.972924
Uniform long-tail	1,000	0.996000	0.977408	2.61900	0.958842

These are deterministic synthetic sessions over non-public catalog targets. They are not organizer-private scores and make no claim about the private distribution. They provide an additional check that ARC's final policy generalizes beyond the 200 visible public labels.

Language robustness

Across 100 public targets, with the reveal wording perturbed:

Robustness condition	Hit@10	MRR	TechnicalScore
Natural paraphrase	1.000	1.0000	0.980600
One hidden clue	1.000	0.9079	0.951775

All audited traces stay within ten turns, return valid unique catalog ASINs, do not repeat proven misses before an override, report zero tokens, and leave the catalog byte-identical.

Runtime disclosure

Resource	Measurement
Agent startup/index build	11.65 s
Mean evaluator wall time per response	38.74 ms
Evaluation wall time after startup	15.34 s
Peak evaluator + agent resident memory	465,644 KB
Prompt / completion tokens	0 / 0
External API calls	0
Estimated inference cost	$0.00
Network / GPU required for scoring	No / No