FangryBirds Shopping Copilot

Repository: https://github.com/rayyngg/FangryBirds-tiktok-tech-jam Demo video: https://youtu.be/Wkq7LaqMDVI Track 4: Shopping Copilot, AI Conversational Search and Recommendations

Inspiration

TechJam's Track 4 hands you a 50,000-product Clothing, Shoes and Jewelry catalog, a simulated customer with a hidden target product, and ten turns to find it. Every turn you can ask about one attribute and show up to ten products. The score is 0.5 times hit rate, plus 0.3 times mean reciprocal rank, plus 0.2 times how early you hit.

The starter kit is a stateless BM25 search that re-queries the catalog with whatever the customer just said. It scores 0.107. Our first attempt added a sentence-transformer for dense retrieval and an optional OpenAI call for wording, and got to 0.67, which felt good for a day. Then we sat down and read the evaluator properly, and realised we had been solving the wrong problem.

The insight that changed the design

The simulated customer is not a person; it is a small deterministic function of the target product. The hidden "intent card" is the product's own metadata: a material word and a colour pulled out by regex, then its feature bullets and detail fields, at most four lines (a budget line exists in the generator but almost never makes the cut). When you ask about an attribute, the customer replies with the next two undisclosed card lines of that class, word for word. The opener names the product's coarse category.

So the constraints the customer gives you are not descriptions of a product. They are strings that exist in some product's record. A customer who says "For that, what matters is: polyester; 75% Polyester, 20% Rayon, 5% Spandex" has just handed you two exact keys. Re-embedding that sentence and doing nearest-neighbour search throws that away.

That reframing is the whole project. What we ended up with is not a better search engine; it is a state machine that keeps a structured record of what the customer has said, mirrors what the simulator believes has been disclosed, and decides each turn whether to ask or to show.

What it does

Each turn goes through five steps.

Parse the message into structured state: category, hard and soft constraints as verbatim strings, a budget window, the set of things the customer has already disclosed, and what we have already shown them. Templates are matched by anchored prefix rather than "everything after the first colon", because product features contain colons, periods and semicolons all the time. Messages that match no template fall through to keyword detection and free text.

Rank a bounded candidate pool (never a full catalog scan) with a lexicographic key: right category first; then how many of the known constraints appear in the product's own intent card, which we reconstruct for every product with a replica of the simulator's rules; then whether the product would have produced the replies we have seen; then plain substring matches; then price within budget; then "not shown yet"; then popularity. These are tiers, not filters. A mis-parsed constraint demotes the target instead of hiding it, and the list is never empty.

Decide how many to show. This is the part we are most pleased with. Per session the score is roughly 0.30 divided by rank, minus 0.02 per turn, so a rank-1 hit next turn is worth more than a rank-2 hit now. The agent estimates the expected reciprocal rank after one more question and only shows rank r if showing it now beats waiting. In practice it shows its single best guess while it is still unsure, says so, and shows the full ten once the answer pins the target. That took the public score from 0.914 to 0.979 with hit rate unchanged at 1.000.

Ask. We ask "other" every turn and never null. That sounds lazy, but under the reveal rule "other" is the only question that cannot come back empty while something is left to reveal, and at most two constraints arrive per turn whatever you ask. We built a proper question-value estimator (expected next-turn reciprocal rank per attribute); it never beat "other" on the public set, so it is kept as an ablation and reused as the deferral estimate in the step above.

Phrase a short templated message. An optional OpenAI call can rewrite it but never touches ranking.

Intent overrides keep the earlier preference (it is still true of the target; the customer just stopped caring about it), promote the new requirement to a hard constraint, and reset the "already shown" bookkeeping, since pre-override turns are never scored.

Being honest about the simulator

Everything above leans on the public simulator's rules, and the private set could differ. The organizer's final-evaluation FAQ, published on 1 September, confirms the final set uses the same deterministic templates and card policy, so what follows covers a harder case than the one we will be scored on. Rather than hope, we wrote a perturbation script that changes the customer at runtime (different card generation, lower-cased constraints, paraphrased openers and replies, the override turn moved anywhere from 2 to 6) and re-scores the unchanged agent. Hit rate stays at 1.000 in nine of ten perturbations and 0.995 in the tenth. Paraphrasing costs rank, not hits: the exact tiers stop firing and the substring and popularity tiers carry the session at around 0.90. The confidence gate never scored below the plain full-list variant under any perturbation, and that is the only reason we made it the default.

Results on the public set (200 sessions)

configuration	score	Hit Rate@10	MRR	MTTC
organizer baseline	0.107	0.125	0.068	9.81
our first version (embeddings, optional LLM wording)	0.672	0.795	0.462	4.19
structured state, full lists	0.914	1.000	0.749	1.53
plus confidence gate (default)	0.979	1.000	0.995	1.97

Zero misses in every scenario. Per-turn latency is about 20 ms mean and 50 ms p95 on a laptop CPU; index construction takes about 5 s. Token usage is 0 by default. A fresh git clone into an empty virtualenv with the network off reproduces 0.979 through the organizer's harness; we script that check and keep the output in the repo.

How we built it

Development tools. Python 3.12 (also tested on 3.9), VS Code, git and GitHub with a pull-request branch for the rebuild, macOS on an M4 Pro and Windows. We used Claude Code as an AI pair programmer for implementation, the evaluation wrapper, the perturbation harness and the test suite; every design decision and every number in this description was checked by us against the evaluator source and the data. Everything is CPU-only.

APIs. None are required. Two optional tiers exist and are off by default: OpenAI gpt-4o-mini for rephrasing the customer-facing sentence (about 150 prompt and 40 completion tokens per turn, roughly $0.00005), and Hugging Face's sentence-transformers/all-MiniLM-L6-v2 loaded strictly from local files for a dense re-scoring tier. Neither improves the public score (dense: minus 0.001; LLM: 0 by construction), so both are disabled and the submission needs no credentials and no network.

Libraries and frameworks. The default agent is Python standard library only: sqlite3 with FTS5 for BM25 over the catalog (with a pure-Python token index fallback if the host's SQLite lacks FTS5), re, dataclasses, json, math. numpy and python-dotenv are small conveniences for the optional paths. sentence-transformers with PyTorch, and the openai client, are only needed for the optional tiers. unittest for the 33 fast tests plus a slow public-set regression.

Datasets and assets. The organizer's frozen 50,000-product catalog and 200 labelled public sessions, both derived from Amazon Reviews 2023 (McAuley Lab, UCSD). The organizer's deterministic evaluator, imported unchanged and wrapped with instrumentation. No manually labelled data, nothing scraped, no outside product information. Every number we quote lives in results/-.json with the commit, environment flags and per-session transcripts attached.

Challenges we ran into

Colons. Our first parser took "text after the first colon" as the constraint. Features like "Solids: 100% Cotton; Heathers: 50% Cotton, 50% Polyester" broke it three different ways. Anchored-prefix matching plus a small search over possible semicolon splits (scored by which split a real product could have produced) fixed it, and we now unit-test our disclosure mirror against the evaluator's own disclosed set after every message.

The gate felt like cheating. Showing one item instead of ten looked like gaming the metric. We only kept it after the perturbation study showed it never lost to full lists, and after making sure the message tells the customer we are holding options back to confirm a detail.

Offline for real. "Works without a key" is easy to claim. We made construction never raise, stopped encoding the catalog at start-up, forced the Hugging Face offline flags, and verified in a scripted clean clone with an empty model cache that nothing was downloaded.

What we learned

Read the evaluator before you reach for a model. The two biggest jumps came from treating the customer's words as exact keys instead of a query (0.792 to 0.911) and from ranking the opener's category first with a popularity prior (0.696 to 0.792); the gate, derived from the scoring formula, added the rest (0.914 to 0.979). Embeddings and an LLM, which we had assumed would be the centrepiece, turned out to be worth nothing on this task, and we are glad we measured that rather than shipping them anyway.

What's next

The exact-match tiers are the part that would not survive contact with real customers, who do not quote product metadata. The structure around them would: per-session constraint state, a disclosure mirror, tiers-not-filters ranking, and a list-length decision derived from the scoring rule. Given more time we would put a fuzzy constraint matcher, or an LLM-backed parser used only when no template matches, between the exact tier and the substring tier; use the user profile (it carries no target information in this simulator, but would in production); and add a one-line explanation per recommendation saying which constraint each product satisfies.