Shopilot

Shopilot
Shopilot's performance for the 200 public test cases

Inspiration

Everyone has had this happen. You know exactly what you want, a specific pair of boots, a particular jacket, you type it into a search box, and you get back forty things that aren't it. So you start clicking filters. Size, colour, price, brand. Ten minutes later you've either found it by accident or given up.

The frustrating part is that a human shop assistant would have solved it in one question. A search box can't ask you anything back. It takes your best attempt at describing the thing and hopes.

That's the gap Track 4 is about, and it appealed to us before we'd read a line of the starter code.

The solution came from a decision we almost didn't make. Instead of starting to code, we spent the first hour reading local_evaluator.py, the file that defines the simulated customer. Buried in it is a function called customer_reply(), and it does something we hadn't expected: it only tells you something if you asked a question. No question, no information.

We went back and looked at the starter agent. It sends ask_attribute: None on every single turn. So it gets the same "ask me about one specific attribute" reply ten times in a row and learns nothing at all, then searches a 50,000-product catalog using that non-answer as its query.

That was the moment the whole thing clicked. This isn't a search problem with a chat interface bolted on. It's a problem about buying information. The agent has ten turns and starts out not knowing which of 50,000 products you want. Every question is an experiment to narrow that down, and the entire game is making each one count.

Everything we built after that followed from that one reframe.

What it does

Shopilot finds the exact product someone wants out of 50,000, and it usually gets there in two turns. The trick is that it picks its questions carefully instead of asking whatever seems reasonable.

Here's an actual session from the test set.

Customer: "I'm looking for Basketball Men, but I'm still exploring."

At this point the agent knows almost nothing. No colour, no brand, no budget, no price range. But that one category is enough to cut the catalog from 50,000 products down to about 180, so it starts there. Instead of firing off ten guesses it puts up a single one, and asks a question.

Customer: "For that, what matters is: polyester; Drawstring closure."

Now there's something to work with. The agent goes through its shortlist looking for products whose pages actually contain that text, gives more weight to whichever phrase is rarer across the catalog, and re-sorts.

The right product comes back first. Two turns and it's done.

There are four things it does that the starter agent doesn't.

It asks a question every single turn. The simulated customer won't tell you anything unless you ask, and the starter never asks, so it burns all ten turns learning nothing.

It asks about the right things. We went through every constraint in the public set and counted them up. feature and material account for 88% of what a customer will ever disclose. brand, category and budget account for zero. Not "hardly ever". Zero. We checked the classifier and it has no bucket for brand or category at all, so asking about them can only ever waste a turn.

It remembers what it's been told. The starter rebuilds its search from whatever message just arrived and throws away everything before it. Ours keeps the lot, so each turn sharpens the search instead of resetting it.

It shuts up when it isn't sure. A session ends the instant the right product shows up in your list, and whatever position it was sitting at becomes your score. So blurting out ten guesses early can win you the round at position seven, which is worse than waiting a turn and winning at position one. The agent holds back until it has something to go on.

	Hit Rate@10	MRR	MTTC	TechnicalScore
Official baseline	0.125	0.068	9.81	0.1067
Shopilot	0.995	0.848	2.06	0.9308

It finds the product in 199 out of 200 sessions, in an average of two turns rather than ten. That works out to 8.7 times the baseline score.

The whole thing runs on the Python standard library. No API calls, no GPU, no network, nothing to pay for. A full 200-session evaluation takes 0.6 seconds.

Why this matters beyond the benchmark

The metric we optimised hardest for is MTTC, the number of turns it takes to reach the right product. It's easy to read that as a leaderboard number, but it's the one with the most obvious real-world meaning.

Every extra turn in a shopping conversation is a chance for the customer to give up. Search abandonment is one of the biggest sources of lost revenue in e-commerce, and the usual response is to add more filters, which pushes the work back onto the customer. Going from ten turns to two isn't a 5x improvement in a score. It's the difference between a conversation someone finishes and one they abandon.

There's also a cost argument. Most conversational shopping tools call a language model on every turn. At scale that's a real bill, it adds latency to every message, and it means an outage at your provider takes your search down with it. Ours answers in 0.2 milliseconds, costs nothing per query, and has no external dependency to fail. For a platform running millions of sessions a day, that difference is not small.

And the thing we actually built isn't specific to shopping. Underneath, it's an agent that works out what it doesn't know and asks the question that narrows it down fastest. That's the same shape as troubleshooting a device, triaging a support ticket, narrowing down a medical symptom, or booking travel. Anywhere someone knows what they want but can't put it in a search box, and where asking the wrong question wastes everybody's time.

How we built it

We ended up with three layers, and the rule we set ourselves was that each one had to still work if you removed the layer above it.

Layer 1 is the whole thing really. It's about 300 lines of Python with no libraries at all, not even numpy. We'd planned 2 and 3 as upgrades but the plain version turned out to be good enough on this benchmark, so we've written up what we actually measured rather than bolting on components that didn't move the number.

Each turn the agent does five things:

Reads the customer's message and pulls out any constraints
Adds them to what it already knows about this session
Filters the catalog down to the shelf that was named in turn one
Scores every remaining product on how well it matches, then sorts
Decides how many products to show, and what to ask next

Step three is doing a lot of work. The category the customer mentions at the start is worked out from the target product's own taxonomy, so the answer is guaranteed to be on that shelf. There's no risk in filtering to it, and it takes 50,000 products down to about 184. It also makes the whole evaluation run 379 times faster, which mattered more than we expected.

Step four is exact phrase matching, weighted by rarity. The simulated customer quotes text straight off the product's page, so if it says "Shaft measures approximately 8.37 inches from arch" then that string probably appears on exactly one product out of 50,000. We build a word frequency index over the whole catalog when the agent starts up, so working out how rare a phrase is costs nothing at query time.

The speed is what made everything else possible. A full 200-session run takes 0.6 seconds, so we could try an idea, get a number, and move on. We got through around 70 experiments over the weekend. If we'd gone with an LLM API we'd have managed maybe five, and we'd have been guessing about the rest.

Everything tunable lives in one config file, and we wrote sweep scripts that try every combination and print the winner. That's how the final settings were chosen. None of the numbers in there are guesses.

The decision that took us longest: whether to use AI at all

Our first instinct was that the answer would obviously be an LLM. AI is in everything right now, it's what we reach for by default, and "conversational shopping agent" sounds exactly like a problem you solve by calling GPT-4o and letting it do the understanding. We spent a good while planning around that before we wrote anything.

Two things changed our minds.

The first was buried in docs/submission_rules.md. It says official scoring may run with network access disabled. If that happens and your agent depends on an API, every call fails and you score zero across all 800 private sessions. Not a lower score. Zero. That alone made an API-dependent design a bad bet.

The second was more interesting, and it only became clear once we'd looked at the data properly. The simulated customer doesn't describe products in its own words. It quotes text straight off the product's page. So it says things like "Shaft measures approximately 8.37 inches from arch", which is a string that appears on maybe one product in 50,000.

An embedding model takes that sentence and turns it into a fuzzy "this is boot-ish" vector that matches every boot in the catalog. A plain substring search finds the one product whose page literally contains those words. For this problem, the AI approach would have been genuinely worse, not just unnecessary.

That was an uncomfortable thing to accept when everyone's instinct, including ours, is to reach for a model first. But we'd rather report what the measurements said than ship something impressive-sounding that scores lower.

We kept the layered design so a model can slot in where it would actually earn its place. If the customer paraphrased instead of quoting, or if two products had near-identical descriptions, Layers 2 and 3 are exactly what you'd want. Neither of those is true here, so we left them off and said so.

Proving each part earns its place

We measured every component by removing it and re-running.

variant	HR@10	MRR	MTTC	Score
Full system	0.995	0.848	2.06	0.9308
minus asking questions	0.795	0.476	4.45	0.6712
minus shelf filter	0.910	0.720	3.23	0.8262
minus cross-turn memory	0.930	0.778	2.92	0.8602
official baseline	0.125	0.068	9.81	0.1067

Asking questions is worth a quarter of the entire score on its own. Every number here is reproducible with one command from the repo.

Checking it wasn't just fitted to the practice data

The 200 public sessions are what we tuned on, so they can't tell us whether the thing generalises. We built our own held-out sets from catalog products that never appear in the public 200, at the same scenario mix.

held-out set	how we sampled it	Score
realistic (n=800)	popular products, matching how the public set was drawn	0.9379
adversarial (n=800)	uniform random, including obscure long-tail items	0.8678
public (n=200)	the set we tuned on	0.9308

We score slightly higher on 800 products we've never seen than on the 200 we tuned against. Two misses out of 800. Our honest expectation for the private set is 0.90 to 0.93.

Challenges we ran into

This being our first hackathon, it definitely wasn't smooth sailing.

We've both done software projects in school, so we thought we knew roughly what we were in for. It turned out to be a different kind of hard.

In a school project you usually get to define the problem yourself. If something turns out to be difficult you can quietly adjust the scope, and as long as the thing runs and you can explain it, you've done the assignment. Here the problem was already fixed, the goalposts were set by someone else, and there was a number at the end that didn't care how much effort went in. You either beat 0.1067 or you didn't. That's a very different feeling to sit with.

The other thing we weren't ready for was how much of the work happens before you write any code. We spent our first proper session reading someone else's codebase instead of building anything, which felt unproductive at the time and turned out to be where every real idea came from. We had to understand the evaluator, the data format, the scoring formula and how the simulated customer behaves before a single line we wrote would have been pointed in the right direction. Nobody teaches you that in a module. You just have to sit there and read until it makes sense.

Then there was the time. About a week from getting the problem statement to submitting, and only three days of that were the actual build window. Learning how something works, working out what to build, building it, then testing and writing it all up, all inside that. There were plenty of moments where we weren't sure we'd have anything to show.

We also had to get comfortable with cutting things. We had ideas we never got to, and a whole layer of the architecture we designed and then deliberately left switched off because it didn't improve the score. Choosing not to build something is harder than it sounds when the clock is running and you want the project to look impressive.

We had one of our main ideas completely backwards.

Early on we built the agent to return all ten recommendations every turn. It seemed free. There's no penalty for guessing, so why not fill the list?

It turns out there is a penalty, and it's a big one. The session ends the moment the right product appears anywhere in your list, and whatever position it was sitting at becomes your score for that session. So if you throw out ten guesses on turn one and the right answer happens to be seventh, you've "won" and locked in a terrible score, and you never get the chance to do better.

Ranking is worth 0.30 of the score. Each extra turn only costs 0.02. So waiting two turns to win properly beats winning badly straight away, by a lot. Once we made the agent hold back when it wasn't confident, the score went from 0.8698 to 0.9211. That's the single biggest change we made all weekend, and it came from making the agent slower.

We caught ourselves nearly cheating.

Something didn't add up. Browsing sessions were getting solved on turn one, but a browsing customer hasn't said anything useful by then. With no information the agent is basically guessing, so how was it winning?

We went and looked. 135 of the 200 target products sit in the first ten items of their shelf, out of shelves averaging 275 products. That's not a skill, that's an artefact of how the dataset file was put together. We were getting free wins from the order of a text file.

Two things bothered us about that. If the private set was built differently the score would fall apart and we'd have no warning, and if a judge found it before we did it would look a lot worse than if we found it ourselves. So we shuffled every shelf into random order and ran the whole thing again. About 4% of our score comes from catalog ordering. The other 96% is genuine matching. We've written that number into the README rather than leaving it for someone else to discover.

Knowing when to stop.

The last day was mostly failure. We ran four more rounds of tuning and every one came back with nothing. Coverage-based sorting, length normalisation, punctuation-insensitive matching, margin-based confidence, title weighting, using the customer profile. All neutral or worse.

The strangest one was a genuine bug. The harness builds product text from a details dictionary as "key value" but builds the customer's constraints from the same dictionary as "key: item", so "Material: alloy" could never match "material alloy". We found it, fixed it, and the score went down. Those details-based constraints turn out to be the generic ones, so the bug had been quietly filtering out useless phrases for us the whole time.

Four rounds of nothing was frustrating at the time, but it's how we knew we'd hit the ceiling of a text-matching approach. That's when we stopped optimising and started writing everything up, which was probably the right call.

Accomplishments that we're proud of

8.7 times the baseline, with no dependencies, no network and no cost.

Browsing sessions went from 2.5% to 100%. That's 40% of the test set that was scoring almost nothing before we touched it.

It generalises. 0.9379 on 800 products it had never seen, slightly better than the set we tuned on.

We found our own dataset artefact and measured it rather than hoping nobody noticed. 4% of the score comes from catalog ordering, 96% is real.

We kept the six ideas that failed and wrote down why each one didn't work.

And underneath all of that: this is our first hackathon, and we finished with the tests done and the writeup written rather than scrambling at 3am. Not just that it works, but that we can explain why every part of it is there.

What we learned

Read the thing before you build the thing.

Every idea that actually worked came out of the hour we spent reading local_evaluator.py instead of coding. Not one came from brainstorming beforehand.

That was a real shift for us. In school we'd always start by getting everyone in a room and talking through solutions. Here the useful first move was the opposite: run the baseline agent, see exactly where it falls apart, and work out what we were actually up against before proposing anything. The customer only answers when asked. The category mentioned in turn one is derived from the answer itself. The constraints are quoted word for word off the product page. All of that was sitting there in plain Python the whole time. We nearly skipped it because reading someone else's code felt like procrastinating.

The scoring formula is a set of instructions, not a scoreboard.

We treated 0.50 x HitRate + 0.30 x MRR + 0.20 x Efficiency as something to check at the end. It's actually a description of what the system is supposed to do. Once we sat down and worked out what each metric was worth in practice, it told us straight away that ranking beats speed by more than ten to one, which is the opposite of what we'd assumed. That one bit of arithmetic was worth more than any feature we added.

Being able to test fast is worth more than being clever.

A full evaluation takes 0.6 seconds, so we could have an idea and know within a minute whether it helped. We got through around 70 experiments that way. The version of us that built an LLM pipeline would have managed five, waited two hours for each one, paid for the privilege, and still been guessing about the rest. We didn't appreciate how much of an advantage that was until we were well into the weekend.

Experiments that fail are still results.

Our last day was almost entirely failure. Six ideas tested, none helped, two made things noticeably worse. At the time it felt like a wasted day. But four rounds of nothing is exactly how we knew we'd hit the limit of a text-matching approach, and it's why we stopped tuning and started writing instead of grinding at a number that wasn't going to move. We've kept every one of them in the writeup rather than only reporting the things that worked.

Reaching for AI is a reflex now, and reflexes are worth questioning.

We assumed from the start that the answer involved a language model, because at the moment everything does. Working out that it would genuinely have scored worse here was uncomfortable to accept. It's probably the most useful thing we're taking away. The interesting question isn't whether you can add AI to something. It's whether the problem in front of you actually needs it.

What's next for Shopilot

Make it work with people who don't quote product pages.

This is the honest limitation and it's the first thing we'd fix. The simulated customer describes what it wants using text lifted straight off the product's listing, which is why exact phrase matching works so well here. Real people don't do that. Someone says "something warm for hiking" rather than "Thermolite insulation, shaft measures 8.37 inches". Against actual language our matching would fall over fairly quickly.

That's exactly the gap Layers 2 and 3 were designed for. A small local model that understands "warm for hiking" and "Thermolite insulated winter boot" are the same idea, and one that can pull structured constraints out of a messy sentence. We built the architecture with slots for both and left them switched off because they didn't help on this benchmark. With real users they'd stop being optional.

Model uncertainty properly instead of approximating it.

At the moment the agent decides how many products to show based on the turn number and how many clues it has. It works, but it's a stand-in for the real thing. What it should be doing is keeping an actual probability distribution over the candidates, measuring how concentrated that distribution is, and committing when the uncertainty drops below a threshold. We built and tested a version that used the score margins and it scored worse (0.9132 against 0.9308), so we didn't ship it. A proper posterior with entropy-based stopping is still the right way to do this and we'd want to give it more than an afternoon.

Handle the cases where two products are basically the same.

Our remaining failures are almost all one thing: a plain product on a crowded shelf, where the customer's clues are shared by forty other near-identical items. No amount of text matching separates those. It needs either a model that can pick up finer distinctions, or the agent recognising it's stuck and asking a genuinely different kind of question.

Built with, and what it costs to run

Language: Python 3.10+

Libraries and frameworks: none. Standard library only (re, json, math, pathlib, collections). requirements.txt is empty.

APIs: none. No external service is called at any point.

Models: none. No language model, no embeddings, no pretrained weights.

Development tools: VS Code, Git, GitHub. We wrote our own measurement scripts for ablation, parameter sweeps, robustness testing and held-out validation.

Datasets: the organiser's frozen 50,000-product catalog (Amazon Reviews 2023, McAuley Lab, UCSD) and the 200 labelled public development sessions from the participant kit.