Inspiration

Shopping is not a database query. People start vague — "something comfortable for standing all day" — then add constraints as they remember them, reject options, and change their minds halfway through. Keyword search punishes all three behaviours.

The provided starter agent made that visible: it treated every turn as a fresh keyword search, scoring 0.1067. It had no memory of what you'd said and no way to handle you taking something back.

But the moment that actually defined this project came later. We had an agent scoring 0.87 and we were pleased with it. Then we fed it real Amazon search queries instead of the simulator's phrasing, and it scored 0.04.

Not 0.84. 0.04. Worse than the baseline we had beaten by 8×.

We had not built a shopping agent. We had built a template matcher that was excellent at one specific way of phrasing things. Everything good in this project came from chasing that number down.

What it does

Goated Agent holds a conversation with a shopper and returns ranked products, converging on the item they actually want in a median of under three turns. Three ideas do the work.

Extraction is a cascade, not a parser. Three tiers run in order and stop as soon as one produces something. Templates first. Then a gazetteer we mined from the catalog itself — 107 colours and 107 materials, where a hand-written list carried 12 and 10. Only if both find nothing does gpt-4o-mini see the message, and even then it may only add spans the rules missed; it can never rank products, clear state, or decide when to retrieve.

Changing your mind is an event, not an edit. Every disclosure is appended to an immutable log, and the live constraint set is replayed from that log each turn. So "actually, ignore what I said about leather" supersedes the old preference rather than overwriting it. The old value leaves the active set but survives at 0.4 weight — because in practice, what you first asked for still describes the thing you want.

Hard requirements cannot lose to a fluent match. Ranking runs in ordered stages: hard-constraint coverage first, then lexical relevance, then soft preferences, with the anonymised profile only ever breaking ties. A product that violates your budget cannot out-argue that by matching your words well.

How we built it

We split the agent into three layers with typed contracts between them — extraction, state, retrieval — so each could be replaced and measured independently. That was our architecture and also how five people worked in parallel without colliding.

Then we did the thing that mattered most: we built two more customer simulators to attack our own agent.

The official simulator has the shopper recite the target product's own catalog metadata. Optimising against only that teaches you to match templates. So we added realistic, which paraphrases the same need in ordinary language, and esci, which opens with verbatim queries from Amazon's public ESCI Shopping Queries dataset. Every change was scored against all three, across 1,400 sessions, and a change that helped one while hurting another was treated as overfitting rather than progress.

That harness is what turned the 0.04 into a diagnosis. We traced it to a single line of behaviour: the search query was built only from extracted slots, so when extraction failed on natural phrasing, the shopper's own words were thrown away. BM25 on the raw text recalled the target 82% of the time. The query we were actually sending recalled it 3% of the time — because on 86% of real queries, it was the empty string.

The agent was standing between the shopper and the search engine, discarding the question.

Challenges we ran into

The measurement that inverted a decision. We built the vocabulary miner and turned it on for every attribute, including product category. The real-query score dropped — 0.7268 down to 0.6437. Guessing "bras" for a listing filed under "Lingerie Accessories" earns a −20 penalty that lands on the correct product. Category ships disabled, one environment variable from returning once soft constraints abstain instead of penalising. Everything except category was worth +0.045.

A feature that was quietly destroying constraints. Our negation layer marked "without horns" correctly. It also marked "No Closure closure", "Non-Polarized" and a product description containing "Never Truly Part" — catalog metadata whose own names contain a negation word. Auditing 1,400 messages found six negations, and five were false positives that deleted real requirements. Negation now applies only to short spans the shopper wrote themselves.

Three ranking ideas that measured worse. Cross-turn rank fusion scored 0.7732 against 0.8183 — it entrenched mistakes made when the agent knew least. A constraint-coverage bonus and IDF-weighted evidence were neutral to harmful. All three ship disabled with their numbers recorded in the source, so nobody re-tries them blind.

The pillar we tested and turned off. The problem statement suggests separate buying and browsing tracks. We built it. The control scored 0.8551; full routing fell to 0.7934. We think the reason is structural: "buying" and "browsing" are too coarse a split, and constraint specificity, candidate entropy, and rank separation are better signals. We kept override replay, disabled the routing, and wrote down why.

What we learned

A single benchmark can lie to you, and it will do so encouragingly. The 0.87 felt like progress and was partly an illusion. Building adversaries for our own system was the highest-leverage engineering decision we made.

Negative results are results. Four of the ideas we are most confident about are ideas we turned off. Each one ships with the measurement that killed it — that is cheaper for the next person than a clean repo that hides the graveyard.

Reach for the model last. Every reflex said "add an LLM." The empty-query bug would have been masked by one, not fixed. We do ship an LLM — it is the third tier of extraction — but it contributes about +0.004, a rounding error next to the +0.74 that came from understanding our own pipeline. The model earns its place at the edges, after the engineering, not instead of it.

Graceful degradation is worth designing for, and we got to prove it. Two bench cells lost connectivity mid-run. They fell back to the deterministic cascade and scored identically to our no-LLM baseline. The fallback is verified under a real failure, not merely asserted.

What's next for Goated Agent

An override-aware retrieval reset, since intent_override is our weakest scenario. Widening the LLM gate to cover low-confidence guesses, not just empty ones. And a reranker fitted on the 234 human-labelled ESCI relevance judgments rather than on simulators we wrote ourselves — because fitting to your own generator can only teach a model to imitate it.

How our solution addresses the problem statement

Pillar I — Intent routing and hybrid pipeline. Multi-route retrieval is implemented: BM25 over an in-memory SQLite FTS5 index, a determinate-violation prefilter, and staged reranking, all in-process with no vector database. We also implemented dual-track buying/browsing routing, measured it, and disabled it — it cost 0.06 TechnicalScore against the control. That negative result and its protocol are documented rather than hidden.

Pillar II — Multi-turn scenario evolution. The state machine is event-sourced: incremental slot accumulation, and intent override handled by supersession with replay rather than destructive slot erasure. The agent returns candidates and asks a clarification in the same turn, so a question never costs a turn of value.

Pillar III — Self-evolution. Session state is distilled per turn into active constraints, rejected values, explicit no-preference fields, and weighted lexical evidence. The anonymised user profile is consumed as a final-stage tie-break. Runtime strategy is data-driven — retrieval diagnostics feed the clarification policy — rather than a fixed script.

Pillar IV — Evaluation. Measured on the unmodified official evaluator, and then again on two adversarial phrasings we built ourselves. Coverage, precision and efficiency are reported for every configuration, including the ones that lost.

	HR@10	MRR	MTTC	TechnicalScore	Tokens
Goated Agent, as shipped (Tier 2 on)	0.9550	0.6890	2.855	≈0.847	~18,000
Same agent, no key or no network (fallback)	0.9500	0.6840	2.875	0.8427	0
Provided BM25 baseline	0.1250	0.0680	9.810	0.1067	0

7.9× the baseline TechnicalScore. The shipped row moves by a few ten-thousandths between runs because the model is non-deterministic; the fallback row reproduces exactly. The gap between the two is ~0.004, which is the whole argument of the "reach for the model last" section above.

Development tools used

VS Code — primary editor, with the Python extension
Claude Code (Anthropic) — AI pair programming across the codebase
Git and GitHub — branch-per-workstream with pull-request review, and exclusive file ownership per contributor to avoid merge conflicts
Python 3.13.9 on macOS (Apple Silicon), CPU only, no GPU
unittest — 198 automated tests
Custom harnesses we wrote: tools/bench.py (3 datasets × 3 simulators), tools/trace_runner.py (turn-by-turn transcripts), and scripts/measure_recall.py (the BM25 recall gate that found the empty-query bug)

APIs used

OpenAI gpt-4o-mini — Chat Completions with a JSON-schema-constrained response. On by default, used only as Tier 2 of the extraction cascade: it may add verbatim constraint spans the deterministic tiers missed, and can never rank products, clear state, or decide when to retrieve. The structural gate fires roughly once per two sessions — about $0.003 and ~18,000 tokens per 200-session run. Any OpenAI-protocol gateway works via OPENAI_BASE_URL. Set TECHJAM_LLM_EXTRACTOR=0 to disable it entirely.
No other external API. Retrieval, ranking and dialogue state make zero network calls; only Tier 2 ever leaves the machine.

Libraries and frameworks used

Retrieval, ranking and dialogue state use no third-party libraries at all. That entire core is Python standard library — sqlite3 (FTS5 full-text index over 50,000 products), re, json, gzip, dataclasses. This is enforced, not claimed: the agent is imported with numpy, torch, openai and sentence-transformers blocked at the import hook, and it still runs and scores.

openai — required for Tier 2; the one shipped runtime dependency

Optional, unused by either configuration in the table above:

numpy — the dense-retrieval package
sentence-transformers — catalog embeddings, built and measured and deliberately unused: a miss audit found 100% target reachability in the candidate pool, so a second candidate generator had nothing left to find

Datasets and assets used

Amazon Reviews 2023 (McAuley Lab, UCSD) — the organizer's frozen 50,000-product Clothing, Shoes & Jewelry catalog and 200 labelled public development sessions. Read-only; no ASIN was introduced, replaced or fabricated.
Amazon ESCI / Shopping Queries Dataset (public) — real shopper query text and human relevance labels, joined onto the frozen catalog to build our esci1000 evaluation set (1,000 sessions, 234 carrying human gold judgments). No external data was used to reconstruct unreleased evaluation labels.
data/lexicon.json — our own asset, 908 attribute values mined deterministically from the frozen catalog by scripts/build_lexicon.py (519 category, 107 colour, 107 material, 105 brand, 41 style, 29 size). The file records the catalog SHA256 it was built from.
data/synth_set_800.jsonl — 800 held-out sessions we generated from the frozen catalog to test generalisation. Used for evaluation only, never to fit anything.