Needle

one conversation, two products, one remembered occasion
needle measures which question is worth asking
blue replaces black. it does not get appended to it
price, stars and review volume stay typed all the way through ranking
nike 806580-404 men air zoom structure 19 blue grey/white-racer blue-blu
the extracted submission, measured by the unmodified official evaluator

Inspiration

Every shopping demo starts with a customer who already knows what to type. Real customers do not.

They say, "I'm getting married and need a suit." Then they change the color. Then they ask for shoes. They remember a budget three turns late. They say "I would not mind red," which means the opposite of "I do not want red."

Ordinary search handles this by returning another page of plausible products. Track 4 does not reward plausible. It gives us a frozen catalog of 50,000 products, at most ten turns, and one hidden parent_asin. If that exact product is absent, the conversation failed.

That led to one question: what if a shopping assistant treated conversation as a way to remove uncertainty, instead of a chat box placed above search?

What it does

Needle keeps a ledger instead of a transcript. It records what the shopper wants, what they ruled out, which answer replaced an earlier one, and which item in a shopping journey each preference belongs to.

On every turn, Needle retrieves products that the catalog can support and measures what is still ambiguous. The storefront asks the question expected to shrink that candidate set most. Its decision receipt shows how many products were considered, what was rejected, which evidence survived, and why that question was selected.

Corrections change the search instead of being appended to it. "Not black, make it blue" retires black without accidentally excluding blue. An explicit change of intent creates a new version while preserving answers that still apply.

The same state can span more than one product. A request for a wedding suit can become a request for matching shoes while keeping the occasion, wearer, budget, and selected suit in view. Compatibility claims are tied to catalog evidence. Price, rating, and review-count instructions operate on typed fields rather than loose text. "Best rated" uses review confidence as well as stars, so one five-star review does not automatically outrank a product tested by hundreds of buyers.

How we built it

The catalog does the heavy lifting.

Needle's evaluated path is deterministic, offline, and built with the Python standard library. A catalog-bound SQLite index stores exact signatures and fielded FTS5 search data. Exact evidence is used when it is safe; fielded BM25 retrieval takes over when it is not. Category evidence and a deliberately small catalog-popularity prior resolve weak ties without overriding what the shopper said.

Conversation state is versioned. Constraints carry polarity, turn number, and intent version. Retired constraints remain visible for audit but cannot leak back into retrieval. Negation is scoped to the clause that introduced it.

The benchmark agent and the product storefront are deliberately separated. Tests prevent storefront code, explanations, translations, or shopping-plan logic from reaching the official starter.agent:Agent ranking path. This lets us demonstrate a richer product without quietly changing the system behind the reported score.

The release archive is built as its own environment. It contains the generated index needed for fast startup, verifies that index against the catalog and parser, extracts into an empty directory, and runs the unmodified official evaluator before the zip is accepted. If the index is absent or mismatched, Needle rebuilds an equivalent database in memory.

Complete build disclosure: We used Python 3.10+, the Python standard library, SQLite with FTS5, HTML, CSS, and JavaScript. Pytest was used as a supplementary development test runner. The only runtime API is the official local Python Agent interface; no external API is called. Development tools were Git, GitHub, GitHub Actions, Visual Studio Code, PowerShell, GitHub CLI, Node.js, Google Chrome headless mode, the Chrome DevTools Protocol, and standard macOS command-line utilities (zsh, curl, unzip, shasum, and sysctl). Anthropic Claude through Claude Code and OpenAI Codex assisted with source, tests, review, and release work. OpenAI ChatGPT was used during early ideation and planning. The supplied TikTok TechJam 2026 participant kit and its frozen catalog, derived from Amazon Reviews 2023 Clothing_Shoes_and_Jewelry, were the dataset sources. The only generated runtime asset is our catalog-bound SQLite signature index.

Challenges we ran into

The first hard problem was language that looks almost identical and means the opposite:

"I would not mind a red shoe."
"I do not want a red shoe."
"Anything but red."
"I am not sure, maybe red."

A broad negation rule fails this test immediately. We stopped treating a sentence as one bag of words, scoped polarity to individual clauses, and kept adding adversarial examples until corrections stopped poisoning their replacements.

The second problem was knowing when a better score was lying to us. Some ranking changes improved the released sessions and lost on catalog-disjoint targets or robustness panels. We rejected them. The selected configuration is not the most flattering result we observed; it is the one that survived the strongest checks we could run in the window.

The third problem scored exactly zero. An early clean submission bundle crashed before its first session because the generated index lived under a gitignored development directory. It worked in the repository and failed where it mattered. That bug changed our definition of "done": the archive itself became a tested artifact, not a folder copied at the end.

Accomplishments that we're proud of

The final archive produced these results on the unmodified public evaluator across all 200 released sessions:

Metric	Result
HR@10	1.000000
MRR	0.996667
MTTC	2.025
TechnicalScore	0.978500
Contract violations	0 across 405 responses
Prompt and completion tokens	0 / 0

Three 200-target proxy panels exclude every released target while matching the released set's broad catalog and scenario distributions. The same frozen agent scored 0.979075, 0.965625, and 0.959900 on those panels. They are development measurements, not predictions of the private evaluation.

The repository discovers 589 tests: 586 pass in a source checkout and three asset checks skip until the untracked release asset is present. A second team member independently rebuilt the archive, extracted it outside the repository, and reproduced the official result on Python 3.10 and 3.12.

We are equally proud of what we did not ship. Higher-scoring but brittle priors, unsafe hard exclusions, semantic components without reliable transfer, and features that weakened packaging or latency gates remain rejected in the experiment record.

The source code and reproducibility guide include the technical report, complete disclosure inventory, dated evidence, negative experiments, and the exact commands used to reproduce the result.

What we learned

The difficult part of conversation is not remembering a word like "blue." It is knowing that blue replaced black, belongs to the shoes rather than the suit, and should still matter three turns later.

We learned that a question is part of ranking. Showing more products spends the shopper's attention; a good question earns information.

Exact methods can be the more intelligent choice. In a frozen catalog with a constrained vocabulary, catalog evidence was faster, cheaper, easier to audit, and more reliable than asking a general model to guess. Needle uses no model tokens because the task did not justify them.

Reproducibility turned out to be product work. A result that depends on one laptop, an ignored file, or an undocumented command is not a result anyone else can trust.

What's next for Needle

The Track 4 build is frozen. We would next test the decision loop against conversations we did not design: hand-written ambiguous requests, unfamiliar correction language, cross-lingual preference extraction, and catalogs whose fields change while a session is active.

For a production deployment, we would add per-session privacy controls, observability around failed interpretations, live catalog versioning, and load tests beyond the local concurrent-client harness. A semantic parser is a possible fallback, but only behind the same evidence, latency, cost, and rollback gates as every other component.

Needle's direction stays the same: when search is uncertain, ask something worth answering.