Q2 | Devpost

Inspiration

We were inspired by the challenge of building a recommender system that can improve itself through repeated experimentation. Instead of relying on a single manually designed model, we wanted to create a research workflow that evaluates hypotheses, measures results, and keeps only changes that improve validation performance.

The project focuses on the KuaiRand-Pure short-video recommendation benchmark, where the goal is to rank videos for each user according to their likelihood of generating a long view interaction.

What it does

We learned that improving a recommender system is not simply a matter of making the model larger or adding more features. Because the evaluation is based on within-user ranking, features that are constant for a user do not change the ordering of that user's recommendations.

We also learned the importance of separating training, validation, and hidden test data. All model decisions were based on the training split and public validation results. The hidden test split was not used for tuning.

Our experiments showed that a small increase in L2 regularization improved validation performance, while increasing model capacity and adding a user_id × tab feature did not help. We also found that a simple pairwise ranking implementation can perform poorly if its optimization procedure is not carefully designed.

How we built it

We started with the official NumPy-only Factorization Machine baseline provided in the starter kit. The baseline uses categorical interactions between:

user_id video_id author_id tab video duration bucket

We then created an iterative experiment workflow. Each experiment recorded its hypothesis, code modification, validation metrics, test metrics for reference, and final decision.

The main experiments included:

Learning-rate tuning L2 regularization Adding a user_id × tab cross feature Replacing pointwise log-loss with pairwise BPR-style training Modifying the pairwise gradient updates

The best retained configuration used the original pointwise FM objective with stronger L2 regularization:

l2 = 1e-5

This achieved a validation primary score of 0.6025, compared with 0.6015 for the original configuration.

The final submission was generated in the required CSV format:

row_id,user_id,video_id,score

and validated using the official submission checker.

Challenges we ran into

The main challenge was improving an already strong baseline. The official FM model had already captured much of the useful signal from the user–video interaction data, leaving only a small amount of remaining improvement.

Another challenge was aligning the training objective with the evaluation metrics. The benchmark evaluates ranking quality using GAUC and nDCG@5, while the original model is trained with pointwise log-loss. Our initial pairwise experiments performed significantly worse because the simple SGD implementation was unstable and did not effectively optimize the ranking task.

We also had to carefully manage the development environment and dataset. The project was initially located inside OneDrive, which caused synchronization issues when the Python virtual environment generated hundreds of files. Moving the working copy to the Ubuntu filesystem resolved this problem.

Overall, the project demonstrated that systematic experimentation, careful validation, and reproducible logging are often more valuable than blindly increasing model complexity.

Accomplishments that we're proud of

We successfully reproduced the official KuaiRand-Pure Factorization Machine baseline using only the training and validation data. We also built a reproducible experiment workflow and evaluated multiple possible improvements, including learning-rate tuning, L2 regularization, additional cross features, and pairwise ranking loss.

Our best retained configuration increased the validation primary score from 0.6015 to 0.6025 by strengthening L2 regularization from 1e-6 to 1e-5. We also maintained experiment records showing which ideas were retained or rejected and why.

Another accomplishment was building a complete submission pipeline that generates the required CSV format and validates row alignment, row counts, and score values before submission.

What we learned

We learned that recommender-system performance depends not only on model complexity, but also on how well the training objective matches the evaluation metric. Increasing the embedding dimension and adding static features did not improve performance because the original user–video interactions already captured much of the available signal.

We also learned the importance of avoiding data leakage. Model decisions were based on the training split and public validation results, while the hidden test set was not used for tuning.

Our pairwise-ranking experiments showed that changing the loss function requires careful optimization. A simple SGD implementation produced unstable ranking performance, even when the training loss appeared to improve.

What's next for Q2

In Q2, we plan to develop a more complete autonomous research loop. The system will generate experiment hypotheses, modify or configure the training pipeline, run evaluations, compare validation results, and record the outcome automatically.

Planned improvements include:

Implementing a stable pairwise or listwise ranking objective with mini-batch training and Adam optimization Adding time-aware user-history features without leaking future information Exploring multi-task learning using signals such as clicks, likes, follows, and watch time Adding automatic retry and recovery for failed experiments Producing structured experiment logs containing hypotheses, code changes, metrics, and resource usage Comparing stronger models such as DeepFM or DCN after the simpler methods are properly evaluated

Our longer-term goal is to reduce manual intervention and create a robust agent that can continuously discover and validate improvements to the recommendation pipeline.