Terminal: local eval baseline (rules/BM25, offline). Sampled sessions
Terminal: Local eval metrics
Terminal: DeepSeek LLM intent run: 1 session, HR@10=1.0, MRR=1.0, MTTC=2.0, tech_score=0.98, 4.4s.
Terminal: Intent eval details: 1 buying session, tokens 1729, tech_score 0.98.
main_agent.py: dialogue routing, BM25/hybrid retrieval, candidate signals, rerank decision logic.
main_agent.py: reranker invocation, question decision, and response emission logic.
main_agent.py: builds response, records tokens/usage, commits dialogue decision and logs session.
Shopping copilot chat
Engineering metrics panel
Eval dashboard overview
Official 200 eval results

Shopping Copilot

TechJam2026 · AI Conversational Search & Recommendation

Inspiration

We started from a feeling everyone recognizes: you type a perfectly reasonable query into a shopping search box and get 40 pages of items that are almost right. Somewhere in there is the thing you actually want — the box just won't ask you what that is.

TechJam2026 made that feeling into a crisp game: a simulated customer knows exactly what they want, and the agent has 10 turns to find it — by asking the right questions and ranking the right candidates. The catch is the information asymmetry: the customer only reveals what they choose to, and the target is never announced.

That asymmetry changed our whole mindset. This is not a "retrieve harder" problem — it's a "ask better, then commit with confidence" problem. And the official score told us exactly what to optimize: half the credit for simply finding the item, another big slice for ranking it first, and the rest for doing it fast. Every design decision we made traces back to that one sentence.

What it does

Shopping Copilot is a polite, tenacious little shopping assistant that finds the hidden item in under 10 turns — and it does it fully offline, with zero API cost. It handles all four official customer personalities:

Buying — a concrete requirement up front. We latch onto it and filter the 50k catalog with surgical precision.
Browsing — the customer starts vague. We ask the highest-value question first (literally "anything else?"), which we proved shrinks the candidate pool from 4,930 items to about 300 per turn while keeping a 99% hit rate.
Intent Override — "actually, forget my earlier preference." We version the intent, quietly demote the old preference, and re-rank around the new one.
Boundary — "I don't have a preference for X." We understand that no preference is information too, and stop asking before wasting a turn.

On the public set it scores HitRate@10 = 1.0, MRR = 0.9364, MTTC = 2.33, TechnicalScore = 0.9543 — about nine times the weak BM25 baseline, at zero inference cost, on any CPU, with no GPU and no internet.

How we built it

We read the data before we wrote the code

Our first full day was not engineering — it was inventory. We opened the frozen catalog and the 200 public sessions and let them teach us:

79% of products have no price. So budget can never be a hard filter. Any design that pretends otherwise is wrong by construction.
The hidden target is generated from its own product metadata. Which means the target text satisfies every disclosed constraint at once, while distractors only satisfy pieces. That single observation became our ranking superpower.
Asking "anything else?" is the cheapest information in the game. One turn, up to two arbitrary constraints, pool cut by 94%.

We designed around one number: the score equation, in plain words

The official score is roughly: half for finding the item at all, 30% for ranking it first, 20% for doing it fast. So we didn't chase model size — we chased decisiveness. Ask until you're confident; then commit hard; never waste a turn on a question that won't change the answer.

Four pillars, told like a story

1. Route the conversation, not just the query. Every turn we decide: is this a buying customer (precision) or a browsing customer (diversity)? Then we fuse four retrieval routes — weighted keyword search, category, hard constraints, and a pre-computed semantic channel — into one ranked pool, using Reciprocal Rank Fusion so no route can dominate the others.

2. A real memory, not a chat log. We built a small dialogue state machine that remembers constraints across turns, versions intents, handles overrides, and decides when to stop asking. It asks a question only if the answer would actually change the ranking — which is why our average session ends in 2.33 turns instead of the baseline's 9.81.

3. The agent that chooses its own tools. Our signature design: the agent probes its own environment at startup (GPU? BLaIR vectors? LLM key? network?), then picks the best strategy for this machine from a data-driven lookup table. If a component fails mid-session, it degrades gracefully — hybrid falls back to keyword, rerank falls back to rules. And if it senses the rules are missing paraphrased disclosures, it upgrades itself to use the LLM on the spot. It never silently does worse; it adapts.

4. Ranking that converts — our three "aha" moments.

Full-hit super-linearity: because the target satisfies all constraints at once, we reward items that "complete the whole picture" with a bonus that grows faster than linearly with the number of constraints they satisfy.
Rarity is confidence: we count, over the entire catalog, how many products satisfy all active constraints. If only one product does — that's the one.
Confidence-gated honesty: a hit locks your rank forever, so at low confidence we show fewer recommendations and only open the floodgates when we're confident or the clock is running out. We literally converted early rank-7 hits into confident rank-1 hits.

Challenges we ran into

The hardest thing we did was turn off impressive models

We integrated qwen3-rerank, BGE, and RexReranker — real, powerful models. Then A/B testing told us the truth: as final rerankers on this deterministic evaluator, they lower the score. The lesson hurt: an enhancement that can't beat the baseline on held-out data should not ship. So they stay wired and tested as opt-ins — but the default is the boring, brilliant rule chain that wins.

Paraphrase is our biggest vulnerability — so we stress-tested it ourselves

The public set uses fixed templates; the private 800 may not. Our own stress test showed rules collapse to 0.383 under natural-language rewrites, while adding LLM intent recovers 0.886. We answered with layers — cue-word upgrades, cascaded LLM, paraphrase assets, and a rewrite guard — and we documented the honest trade-off instead of pretending one magic switch fixes everything.

Five teammates, three sources of truth

With five of us contributing, tuning values, config profiles, and runtime defaults drifted apart. The fix was config-as-data: one source of truth, validated everywhere, plus per-session structured logs so every decision and every fallback is auditable.

Accomplishments that we're proud of

~9× the baseline score, at zero cost — pure rules + SQLite reproduce it on any CPU, fully offline.
100% hit rate across all four scenarios — including Intent Override, the one most agents trip on.
2.33 turns vs 9.81 — we ask less and convert earlier; that's the 20% "fast" slice of the score, earned honestly.
An agent that decides what it can use — the environment-aware autonomy layer is the thing we'd defend in any room: it probes, selects, degrades, and upgrades by itself.
We said no when we had to — shipping fewer flashy models than we built, and documenting the A/B evidence for every choice.
Better ranking and tighter output control — ablation tests clearly show the real value of each optimization: the emit gate delivers a significant gain, while fingerprint matching also makes a clear positive contribution.
119 out of 120 new targets — on our frozen new-target set, the agent achieved a 0.992 Exact Hit Rate, showing that the strategy transfers beyond the public targets.

What we learned

Measure before you model. Three free observations about the data outperformed every model swap we tried.
On a deterministic evaluator, decisions beat fluency. The wins came from when to ask, what to emit, and how to rank with confidence — not from bigger models.
Robustness is layers, not a switch. One layer always breaks; a system of layers survives.
Environment-awareness is a feature, not plumbing. An agent that knows its own hardware and network — and picks its strategy accordingly — is something you'd actually trust to run on your machine.
The best feature we shipped might be "saying no" — to the model that didn't earn its place, and to the turn that wouldn't change the answer.

What's next for Shopping Copilot

Adversarial training of a different kind — an automated opponent that mines worst-case messages (paraphrases, partial overrides, ambiguous mentions) and we tune against them.
Learn profiles across sessions instead of treating the profile as a weak prior.
Make the dense channel cheaper so hybrid retrieval runs happily on CPU-only machines.
Explain our recommendations — tell the customer which of their stated needs each item satisfies.
Close the remaining gaps — fix the last recall miss, improve rank-one accuracy, and handle intent changes faster.

Built by a 5-member team · TechJam2026 Shopping Copilot · Full engineering detail in README.md, docs/TECHNICAL_REPORT.md, docs/cost_disclosure.md, and docs/demo_session.log.