SpiderRank

Inspiration

We mainly gained inspiration via research from papers online. We also did quite a bit of brainstorming with Claude for novel ideas, like removing items already shown, and repelling shown items. Our workflow is also inspired by our work in research labs, where we would try out many ideas, iterate over them, and keep only the ones that work. This has proven to be successful in our project. As far as conversational recommendation goes, the general architecture is also based on a RAG type of system, with of course other ranking metrics and rank fusion. Elements like nearest vector search and rerankers are from RAG.

What it does

SpiderRank talks a shopper through finding one specific, hidden product in a 50,000-item catalog in 10 turns or fewer. Each turn it returns ten ranked recommendations and one clarifying question, chosen to be both informative (splits the current candidate pool) and answerable (this particular shopper is likely to actually respond). Underneath: five independent retrieval legs vote and fuse into one ranking, disclosed attributes boost candidates without ever deleting one that might still be right, and a router detects buying vs. browsing vs. a shopper changing their mind mid-conversation. It runs CPU-only, fully offline, with zero external LLM calls, and finds the hidden target 94.5% of the time on the released benchmark, typically within about 3 turns.

How we built it

Everything is Python, built around the organizer's fixed Agent interface and unmodified evaluator. We combined five off-the-shelf retrieval Paradigms, BM25, dense cosine via a frozen encoder, exact-phrase matching over SQLite FTS5, pseudo-relevance feedback, and popularity, with weighted reciprocal rank fusion, and treated every architectural choice as a hypothesis to measure rather than an assumption: change one knob, run the 200-sample public set, record the delta, then decide. Intent, pivot, and non-answer detection all run off the same frozen encoder via nearest-prototype similarity instead of three separate trained models.

Our workflow includes brainstorming ideas, and then implementing them, and using the evaluator to generate results. Based on the lift, we would scrap or keep the idea. We did this iteratively to increase the retrieval score.

Challenges we ran into

The biggest challenge that we encountered was not having access to GPU or LLM APIs. This meant that we are constrained to small models, and LLMs cannot drive this project. Agentic implementations were also out of the question. Hence, what we had to do was implement this RAG based approach with the limited compute resource that we had. This also meant that evaluation runs were very long, and iterations were blocked by evaluation runs and compute power. Another challenge that we faced was the small public dataset size. 200 is not enough to train a classifier head, and there is a very high probability of overfitting to the data. It is also very hard to extract insights from the small dataset size from data analysis.

Accomplishments that we're proud of

Our single largest win (+0.084 TechnicalScore, three times any retrieval change) came from a rule with no paper behind it: never re-show a product already shown, because the evaluator only asks for another turn when every item shown so far was wrong. We're proud we found it by re-reading the scoring function rather than reaching for a bigger model. We're also proud of the discipline behind our experiment ledger: we predicted five ideas should work, measured them, and rejected all five when the numbers said otherwise — including a trained intent classifier that hit 98% in testing by memorizing the simulator's two sentence templates, then fell to chance on anything else. Half our ledger is things we threw away, and we think that's the real engineering story.

What we learned

One big idea that we learnt was to analyse the data before putting it into the model. When we were working with user_profile, we were wondering why it did not lift our results significantly. Afterwards we took a step back, and realized that it was exactly because there is more noise than signal in the data, and pursuing this track would be a dead end. Another example of this would be in the boundary cases, where we realised the input did not have any useful information. Hence, we would have to use another metric (popularity) to come up with recommendations. Another big idea was learning how to code with AI. AI has amazing capabilities in code implementation, but if we do not know what is going on, any engineering mistake that it makes will go unnoticed. Hence in this project, we were deliberate in making the engineering decisions ourselves, and ensuring that AI is only a tool for us to implement our solutions.

What's next for TAISP Baddies Track 4

Most of what’s left in this project is just hyperparameter tuning. I think given our limited compute budget, it is the best that we can offer. Another direction of exploration that we want to go down is using multiple agents as planners and executors. With the powers of LLMs, planner agents would be able to reason about what the user wants. It also has world knowledge, so intelligent inferences are possible. This would open a whole new world of possibilities with our retrieval pipeline.