ESearch

Inspiration

The basic recommendation algorithm is based on our intuition and the observation about the dataset. Intuitions:

Products with higher popularity have been bought by more users. For example, one product has 90 buyers, and another has 10 buyers. Then, 90% of the people buy the product with higher popularity. Our intuition is that, people tend to prefer products with higher popularity.
The target product must match the exact category. To be more accurate, completely unrelated items should not appear in the search.
Previously displayed items should not be displayed again, or at a much lower frequency. This is not generally true for a scrolling-style recommendation system. But for a conversational system, this makes sense.
The next attribute to ask is determined by the coverage and information gain. Coverage means how many items have non-null attributes; information gain means how much information will be obtained by clarifying this attribute (measured by normalized entropy).

Remark: due to the design of the evaluator, we found that if directly asking for "others", the evaluator will provide useful unrevealed attributes, and help improve the score. BUT we regard this as a bug of the evaluator and did NOT adopt the strategy.

LLM-based attributes extraction is inspired by https://aclanthology.org/2023.emnlp-industry.55/

We have also adopted other strategies (see the readme for details), but they cannot improve the benchmark due to the design of the evaluator script.

What it does

The agent parses the input from the user / simulated user, and returns a question plus the recommendations for this round. In the interactive mode, the input is parsed by LLM and the question is generated by LLM. In benchmark mode, we use regex to parse and generate fixed-format questions.

How we built it

We use codex to implement our ideas, with some human intervention so that the AI is always on the right track. We rely on a few dependencies, including numpy, openai, and sentence transformers. We tried to make our repo as self-contained as possible.

Challenges we ran into

The biggest challenge is about the evaluation script. It is not well designed and causes us to spend lots of time trying to balance the benchmark score and real-world performance. All of our models are designed to be both strong in the benchmark and strong in real-world application.

To resolve the issue, we used the source Amazon 2023 dataset and generated 1000 samples so that we can know if our solution generalize well. With 200 samples only, it is easy to fall into overfit.

Accomplishments that we're proud of

Our recommendation system performs much better than we expected. V1 achieves a high benchmark score with a simple but elegant design. V2.12 uses more complicated mechanisms and improves the real-world performance without sacrificing the benchmark score. The computational cost of our models are low and all models except V3 do NOT rely on external LLMs.

What we learned

Basics about the design of recommendation system
Basic LLM chatbot design
How different NLP technologies can be adopted in recommendation systems, including embedding and attribute extraction.
How human intuition works in recommendation system

What's next for ESearch

We would like to use a locally deployed light LLM for V3 to measure the performance
We plan to design a better evaluator and a better benchmark for the task
We plan to balance the computational cost and recommendation system performance, and implement a final version of our agent