Lab Rats

• 

  workflow architecture

• 

  ui

Inspiration

We started by reading the evaluator instead of the problem statement.

The simulated customer isn't a language model it's a deterministic function that lifts requirements verbatim from the target listing and releases them one attribute at a time, gated behind a single structured field, ask_attribute. Once we saw that, the problem changed for us. This wasn't semantic search dressed as a conversation. It was information acquisition: one question per turn, and the whole game is choosing which one.

The provided BM25 baseline scored 0.107. We wanted to know how much of that gap was retrieval and how much was dialogue. Almost all of it was dialogue.

What it does

Shopping Copilot finds a customer's hidden target among 50,000 Amazon listings within 10 turns, asking useful questions along the way.

The LLM is compulsory, not an add-on. Three model stages run by default: grounded constraint extraction, semantic reranking of tied candidates, and message rewriting.

Underneath each one sits a deterministic result already computed. The model's output is adopted only if it arrives, parses, and lands inside the turn's 12-second budget; otherwise the floor stands. There's an offline fallback where the state the system reaches when the online path can't complete.

The core is a constraint ledger: every disclosed requirement stored verbatim, never erased. Candidates from a BM25 pool are ranked by how many stated requirements they literally contain, then category, popularity, and a total tiebreak, where the agent then commits to one pick and asks the question that splits the tie.

How we built it

We read the harness first. Three properties fell out: every turn is scored and asked, so there's no show-versus-ask trade-off; ask_attribute is the only information channel; and requirements are verbatim substrings, so a candidate containing all of them is almost certainly the target.

**We built a ladder, one measurable layer at a time.

Ship rule: a change ships only if it clears $\Delta \geq +0.02$, or delivers a robustness gain with no regression. Ideas we liked but couldn't justify stayed in the repo as measured, disabled flags.

Because 200 sessions isn't many, we built 1,000-session holdouts from the upstream leave-last-out split, plus a uniform-random control to isolate popularity skew. The scored retrieval core uses only SQLite FTS5, re, json, and difflib.

Challenges we ran into

The intent-override trap. Our first instinct is to never show the same product twice, scored 0.834 and collapsed override sessions from 100% hit rate to 10%. A hit before the override fires doesn't count, so a long memory shows the target early, gets no credit, bans it permanently, and can never recover. A one-turn memory means a wrong exclusion self-corrects next turn. The forgetting is the feature.

The override isn't a contradiction. The brief calls for "slot erasure", so we built erasure first. Then we looked at what the customer actually says: the replacement is a hard constraint of the same product, and the discarded value was a soft preference of that same product. Erasing throws away true signal. We accumulate instead; which ended up being the opposite of our first design.

Showing fewer items is sometimes correct. When many candidates tie, the right move is to show one, not ten: a wide tie means the agent can't discriminate, so it should commit and ask. Getting that direction right was worth +0.038 MRR.

A budget that was too tight. The first API call in a fresh process spends about 5 seconds on connection setup, overrunning our original 4-second budget so calls succeeded, we were billed, and the results were discarded anyway. Warm calls take 1.5 seconds. We raised the budget to 12.

Accomplishments that we're proud of

The score is reproducible, not just high. It reproduces to six decimal places on a different operating system, Python version, and SQLite version than the one it was developed on, and is bit-identical across different hash seeds.

The offline fallback is verified, fast and free. We injected connection failures and six-second hangs across all 200 sessions on both providers. All 200 per-session results came back identical to the offline reference, with zero exceptions and zero contract violations. An outage costs six network attempts before the circuit breaker latches, then the run finishes offline at no cost.

We measured what we rejected and published it. The vector-similarity route, the LLM reranker as sole ranker, and the long-term preference prior are all implemented, all measured, and all in the repository behind flags.

What we learned

Read the evaluator. The highest-leverage hour we spent was understanding the customer's information channel before writing a line of agent code.

Measure the thing you're about to believe. Cumulative exclusion, slot erasure, showing more items when uncertain, LLM reranking beating a lexicographic sort, every strong intuition we had was wrong in a measurable way. We only found out because every layer was a flag we could flip.

A benchmark's information channel dictates its architecture. Exact matching beats semantic similarity here because the simulator lifts requirements verbatim from the listing. That's a property of this benchmark, not a general truth about shopping and confusing the two would be the real mistake.

Reliability is a design property, not error handling. Because every model call is an override of an already-computed result rather than a dependency, an outage changes wording, not correctness.

What's next for Lab Rats

Real entity linking. Our matcher is tuned to verbatim strings. Against free-form shoppers it needs to resolve "something warm for winter" onto the catalog's actual attribute vocabulary. That's the first thing we'd build.

Neural embeddings for the second retrieval route. Our TF-IDF vector route measured as free but inert; exact attribute strings dominate on this benchmark. A dense-embedding variant is the version we'd expect to matter against real language. However, optimizing it to be as efficient and fast as possible will remain a challenge.

Built With

• agent
• openai
• pytest
• python