Performance
Playground
Problem Formulation
Case Study

ARC — Ask · Rank · Commit

An answerability-aware, evidence-grounded sequential decision agent for conversational shopping

ARC is an offline multi-turn shopping agent built for TikTok TechJam 2026 Track 4.

One hidden product. A 50,000-item catalog. At most ten turns. The shopper starts with incomplete requirements and may decline a question or change their mind.

ARC treats this as a sequential decision problem, not a one-shot search query. On every turn it makes three connected decisions:

ASK the question that is answerable and worth its extra turn;
RANK products using accumulated shopper evidence, not popularity alone;
COMMIT only as many results as the current evidence can safely support.

System	Hit Rate@10	MRR	MTTC	TechnicalScore
Organizer weak baseline	0.125000	0.068034	9.810	0.106710
ARC (submitted)	1.000000	1.000000	1.980	0.980400

These results use the unchanged organizer evaluator over all 200 public sessions. Every one of the 200 targets is returned, and every one of them is returned at rank 1. The submitted runtime uses zero model tokens, zero network calls, no GPU, and only the Python standard library.

Why shopping needs more than retrieval

A retrieval system can contain the right product and still create a poor shopping experience:

A vague request may leave hundreds of equally plausible products.
A high-discrimination question may be useless if the shopper cannot answer it.
Showing ten weak candidates too early can end the session with the target at a poor rank.
Repeating products that were already rejected wastes both turns and trust.
A changed preference can make otherwise correct conversation memory stale.

The challenge metric makes these failures concrete. Hit Rate rewards finding the target, MRR rewards putting it near the top, and MTTC charges for every additional turn. ARC therefore optimizes the interaction loop, not only a retrieval score.

One line of arithmetic sets the whole policy. Because a session is scored at the turn the target first appears, demoting a rank-1 result to rank 2 costs 0.30 × 0.5 / 200, while saving an entire turn earns only 0.20 × 0.1 / 200. One lost rank is worth 7.5 saved turns. Any design that shows more products sooner has to clear that bar, and almost nothing does.

The three decisions

ASK, RANK, and COMMIT are not three disconnected heuristics. They all read the same explicit turn state:

state(t) = {
  catalog shelf,
  observed constraints + confidence,
  exhausted attributes,
  questions already asked,
  products proved wrong,
  whether signature order is still reliable,
  turn number
}

One call to Agent.respond performs the complete loop:

shopper message
  → update persistent state
  → retrieve the shelf and remove proved misses
  → rank every surviving candidate
  → simulate the answers to every legal question
  → choose the question and recommendation-list length
  → return the response and store an audit certificate

The hidden target ASIN and evaluator label never enter this path. Every update comes from the transcript, the frozen catalog, or the fact that the evaluator continued the conversation.

ASK — spend a turn only when it has value

The naive way to choose a question is to find the attribute that splits the candidate pool most evenly. That fails when the shopper cannot answer it. ARC instead asks: if each plausible candidate were the target, what answer would the simulator reveal, and what would that answer do to the scored ranking?

First, planning is bounded to a competitive pool: at most 512 candidates whose score is at least 75% of the current best, with a minimum of ten whenever ten exist. Let that pool be P, with N = |P|.

For every legal attribute a and every candidate p ∈ P, ARC replays p's four-slot intent signature against the constraints already disclosed:

predicted_answer(p, a) = up to two still-hidden signature values of type a
                         or <none> when p has no answer for a

The open question other can reveal the next two remaining values of any type. Candidates that predict the same answer form a group G. Treating each candidate as a counterfactual target gives Pr(G) = |G| / N, so ARC can compute the expected post-answer metric directly, without training a question model:

ExpectedHit(a) = (1 / N) × ΣG min(|G|, 10)
ExpectedMRR(a) = (1 / N) × ΣG (1 + 1/2 + ... + 1/min(|G|, 10))

Those two expressions apply to answerable groups. The <none> group is treated differently: “I have no preference” adds no positive constraint, so those candidates keep their current ranks. ARC never awards a fictitious rerank to an unanswered question.

The final answerability-aware metric value of information is:

AnswerableMVOI(a)
  = 0.50 × (ExpectedHit(a) - CurrentHit)
  + 0.30 × (ExpectedMRR(a) - CurrentMRR)
  - 0.02

0.02 is not an arbitrary penalty: it is exactly the TechnicalScore cost of one additional turn, 0.20 / 10. ARC selects the available attribute with the largest AnswerableMVOI; exact ties use a stable order headed by other.

For example, suppose material divides 12 candidates into four cotton, three polyester, and five <none>. An entropy policy sees a useful 4/3/5 split. ARC also sees that the five <none> candidates will not move at all, prices that branch at their existing ranks, subtracts the turn cost, and may prefer an open other question that the shopper is more likely to answer. The difference is small in notation and large in conversation quality.

RANK — keep explicit evidence ahead of popularity

RANK starts before scoring. The opening message is grounded to the most specific matching catalog shelf; if that shelf is wrong or exhausted, ARC broadens to the complete read-only catalog instead of returning an empty turn. Constraints then accumulate across the conversation with explicit confidence weights.

Each catalog product has a canonical four-slot intent signature derived from ordered features and details, plus detected material, color, and price when visible to the protocol. For one candidate p and observed constraints c1 ... ck, ARC computes three evidence layers:

Lexical(p)
  = normalized sum of confidence-weighted rare-phrase matches

Typed(p)
  = weighted satisfaction of material, color, and budget semantics

Signature(p)
  = weighted mean of:
      1.0  same value in the same canonical slot
      0.7  same value elsewhere in the signature
      0.0  value absent from the signature

The lexical term uses global inverse document frequency, so matching a rare, specific clue matters more than matching “comfortable” or “women's.” Typed matching prevents synthetic clues such as color: grey or budget around $25 from depending on whether that exact serialized string appears in product text.

The signature position bonus is used only while the transcript guarantees canonical reveal order. An attribute-specific answer or an intent override can reveal a later field early; ARC records that transition and falls back to membership-only signature matching rather than trusting stale positions.

The evidence score is:

core(product) = normalized rare-term evidence
                + 0.30 × typed constraint satisfaction
                + 0.90 × canonical signature likelihood

Popularity is deliberately outside core. For candidates already within 0.15 of the best evidence score, ARC permits a bounded catalog prior:

prior(p) = 0.50 × (log(1 + reviews(p)) / log(1 + max_reviews))^1.25
         + 0.50 × clip(rating(p) / 5, 0, 1)

final(p) = core(p) + 0.55 × prior(p)
           only if best_core - core(p) ≤ 0.15

A popular mismatch therefore cannot jump over a less popular product that actually satisfies the shopper's evidence. The prior decides only among already-plausible candidates.

ARC uses a signature weight of 0.9, validated consistently on the public set, the popularity-matched panel, and the uniform long-tail panel. This gives the canonical intent signature enough influence to separate evidence-equivalent products without allowing the bounded popularity prior to dominate intent.

Cold start is treated as its own state. Before the shopper has disclosed a single constraint — 90 of the 200 public sessions open this way — there is no evidence to rank on, and catalog order is not a prior. That one state is ordered by the disclosed aggregate review count. This is not a general popularity bias: the moment one constraint arrives, the evidence ranker takes over. We verified the choice rather than assuming it, sweeping quality mixes, Bayesian smoothing, rating thresholds, and count exponents; none beat a plain review count, and 70 of the 200 targets turn out to be the single most-reviewed product on their shelf.

Finally, ranking is stateful. If the conversation continues, ASINs emitted on the preceding eligible turn become proved misses and are filtered before the next ranking. A true intent override clears that negative history, decays the withdrawn opening preference to confidence 0.5, and prevents evidence from an old shopping goal from trapping the new one.

COMMIT — control output risk, not just relevance

RANK produces an ordering; COMMIT decides how much of that ordering to expose. This distinction matters because the evaluator stops at the first list that contains the target. Returning ten products is therefore not harmless recall: if the target appears at rank 7, the session ends permanently at reciprocal rank 1/7, even if one more answer would have made it rank 1.

While evidence is incomplete, ARC emits only the safest candidate and continues to ask. A response can therefore contain both a Top-1 recommendation and the next clarification question. Once evidence is complete, a unique leader can be committed normally—but an exact-signature tie needs a different policy.

Some products are genuinely indistinguishable: once disclosure is exhausted, a group of catalog siblings can share a byte-identical intent signature, and no further question can separate them. Let V(t, n) be the best expected remaining score at turn t with n indistinguishable siblings. If ARC emits a prefix of length m, each sibling has probability 1/n; a hit at output rank r earns HitRate, MRR, and current-turn efficiency, while a miss removes the entire batch:

V(t, n) = max over m = 1 ... min(10, n) of

          (1/n) × Σr=1..m [0.50 + 0.30/r + 0.20×(11-t)/10]
          + ((n-m)/n) × V(t+1, n-m)

This dynamic program is tiny, deterministic, and solved in memory. It decides whether one rank-1 trial, a short slate, or the full ten has the best expected TechnicalScore over all remaining turns. On an exact utility tie it prefers the larger batch, so it does not delay without a measurable reason.

Continuation supplies the transition: if the evaluator calls ARC again, none of the m emitted siblings was the target, so all m are removed and the next state is (t+1, n-m). From turns 8–10 ARC forces the full available Top-10, providing 30 final slots of Hit@10 insurance. A shortened early batch therefore cannot lose a hit that the previous full-list policy would have found.

Here is the concrete failure this fixes. In public_0099, all four visible constraints are known on turn 3, but the top three products have byte-identical intent signatures. The target is third only because of the bounded popularity tie-break. Committing all three immediately would lock in:

turn 3, rank 3 utility = 0.50 + 0.30/3 + 0.20×(11-3)/10 = 0.76

The planner instead tests the siblings at rank 1. Two continuations refute the first two, and the target is returned at rank 1 on turn 5:

turn 5, rank 1 utility = 0.50 + 0.30/1 + 0.20×(11-5)/10 = 0.92

Spending two turns gains 0.16 on that session. This is exactly the trade the published metric asks the agent to make, and it is derived without seeing the target.

The replay makes all three decisions visible in one sequence:

Turn	New evidence	ASK	RANK state transition	COMMIT
1	Shelf only:`Active Pants Sweatpants`	`other`, with AnswerableMVOI `0.177131`	Cold-start review-count order over 127 products	One product
2	`cotton`; `60% Cotton, 40% Polyester`	`other`, with AnswerableMVOI `0.134631`	Remove the turn-1 miss; evidence ranking takes over	One product
3	`Imported`; `Drawstring closure`	Stop: all four protocol constraints are known	Remove the turn-2 miss; rank the exact-signature siblings	One rank-1 refutation trial
4	Continuation means the prior output was wrong	No redundant question	Add the turn-3 product to proved misses	One rank-1 refutation trial
5	A second continuation proves the next sibling wrong	No redundant question	The target now has a`0.759965` lead over rank 2	Target at rank 1; safe full Top-10

The final result is a turn-5 rank-1 hit with session utility 0.92. More importantly, every transition has a checkable reason: a disclosed constraint, a computed question value, a proven miss, or the finite-horizon output policy. That is what makes ASK, RANK, and COMMIT one auditable controller rather than three independently tuned tricks.

Proving the turn budget is spent optimally

Claiming a policy is good is easy; we wanted to know whether it was finished. tools/turn_audit.py replays the organizer's own loop while capturing the agent's complete ranked list on every turn, then compares what ARC achieved against what a perfect output gate could have achieved.

The result is the strongest statement we can make about the decision layer:

Turns lost to the output gate: 0. Across all 200 sessions there is not one case where the target had already reached rank 1 and ARC withheld it. The target is exposed on exactly the turn it first becomes the best candidate.

The remaining MTTC is therefore bounded by disclosure, not by policy:

Structural bound	Cost
`intent_override` cannot score before its override turn	floor`3.600` turns, ARC achieves `3.667`
`boundary` spends one turn on the “I have no preference” reply	10 sessions, unavoidable
90 of 200 sessions open with zero constraints	turn 1 is a prior, not a ranking problem

We also checked, product attribute by product attribute, whether anything separates the target from the items ranked above it on turn 1 — review count, average rating, feature count, detail count, description length, price presence, title length, and first-availability date. All of them are coin flips. Turn 1 is an information limit, not a modelling failure.

This is why the first-hit distribution is as compressed as it is:

First hit on turn	1	2	3	4	5
Sessions	76	75	28	19	2

Why we deliberately did not put an LLM in the runtime

The challenge permits LLMs, but it does not require one. We chose the smallest system that directly addresses the measured bottleneck.

In this benchmark, shopper messages reveal catalog-derived constraints and a hit is an exact ASIN match. The dominant uncertainty is which useful constraint has not yet been revealed—not how to generate fluent prose.

What this task requires	Why an LLM is not the default solution here
Exact catalog-valid ASIN ranking	Fluent text does not guarantee exact identifier retrieval or ordering
Persistent constraints and overrides	These need explicit, testable state transitions rather than implicit prompt memory
Metric-aware question and output policy	An LLM does not automatically optimize Hit@10, MRR, and turn cost
Reproducible official scoring	Hosted inference adds network, credential, latency, cost, and nondeterminism risks
Grounded explanations	Generated rationale may sound plausible without matching the actual reranker

The intelligence is concentrated in deciding what evidence to acquire, how to update it, and when it is sufficient—not in generating more fluent prose.

This is not a claim that language models are never useful. It is an architectural boundary: use deterministic catalog evidence for the common path, and introduce a small local language or embedding fallback only when an ambiguous Top-N case shows a measured gain. Today the grounded parser achieves 1.00 Hit@10 and MRR on the natural-paraphrase robustness panel, while passing every audited wording variant. This is not an anti-LLM position: we would add a model when it earns its latency and complexity with a measured gain on unresolved language cases. Until then, putting one on the critical path would add operational risk without addressing the dominant failure mode.

How this maps to the Track 4 directions

The organizer's suggested techniques are possible tools, not a checklist. Our design uses the parts that improve the shopping decision and leaves speculative complexity disabled.

Track direction	ARC implementation	Shopper-facing outcome
Buying vs. Browsing routing	Scenario-aware shelf and question routing	Decisive buyers are served immediately; vague browsers are clarified
Hybrid retrieval and reranking	Shelf retrieval plus lexical, typed, signature, and bounded-prior evidence	Exact constraints beat generic popularity
Structured state and intent override	Confidence-weighted multi-turn memory with atomic history reset	Preferences accumulate without trapping the shopper in an old intent
Adaptive clarification	Answerability-aware metric value of information	The agent avoids questions that cost a turn but add no usable evidence
Failure detection and strategy switching	Proven-miss exclusion, refusal-aware pivot, late exact-tie rotation	Rejected products are not repeated and long-tail ties can recover
Indistinguishable-candidate planning	Finite-horizon batch-size optimization with final-turn Hit@10 insurance	Exact intent twins are tested at rank one instead of committed at a weak rank
Cold-start ranking	Review-count prior only before the first shopper constraint	A vague first turn is useful without letting popularity override intent
Low latency and token cost	Deterministic, offline, standard-library runtime	No API outage, credential, GPU, or per-query model cost
Transparent explanations	Evidence certificates and verified minimal counterfactuals	Engineers can inspect why the action and rank changed
Safe personalization	Aggregate profile support exists, but its ranking weight is disabled	Unproven profile correlations cannot override explicit intent

Dense semantic retrieval and profile weighting remain optional extensions. They were not enabled simply to match a suggested architecture; a new component must earn its complexity across public, popularity-matched, and uniform long-tail diagnostics.

Evaluation

Official public set

Scenario	n	Hit Rate@10	MRR	MTTC
Buying	80	1.000000	1.000000	1.487500
Browsing	80	1.000000	1.000000	1.775000
Intent override	30	1.000000	1.000000	3.666667
Boundary	10	1.000000	1.000000	2.500000
Overall	200	1.000000	1.000000	1.980000

The organizer weak baseline needs 9.81 turns on average; ARC needs 1.980, a reduction of 7.83 evaluator turns while raising MRR from 0.068034 to 1.000000.

Target-disjoint diagnostics

The public 200 are the only labels anyone can see, which makes them easy to overfit. We therefore hold ARC to two frozen panels whose targets never appear in the public set:

Diagnostic	n	Hit Rate@10	MRR	MTTC	TechnicalScore
Popularity-matched	800	1.000000	0.984496	2.12125	0.972924
Uniform long-tail	1,000	0.996000	0.977408	2.61900	0.958842

These are deterministic synthetic sessions over non-public catalog targets. They are not organizer-private scores and make no claim about the private distribution. They provide an additional check that ARC's final policy generalizes beyond the 200 visible public labels.

Language robustness

Across 100 public targets, with the reveal wording perturbed:

Robustness condition	Hit@10	MRR	TechnicalScore
Natural paraphrase	1.000	1.0000	0.980600
One hidden clue	1.000	0.9079	0.951775

All audited traces stay within ten turns, return valid unique catalog ASINs, do not repeat proven misses before an override, report zero tokens, and leave the catalog byte-identical.

Runtime disclosure

Resource	Measurement
Agent startup/index build	11.65 s
Mean evaluator wall time per response	38.74 ms
Evaluation wall time after startup	15.34 s
Peak evaluator + agent resident memory	465,644 KB
Prompt / completion tokens	0 / 0
External API calls	0
Estimated inference cost	$0.00
Network / GPU required for scoring	No / No