On the Effectiveness of Sampled Softmax Loss for Item Recommendation
Top Cited Papers
Open Access
- 22 March 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 42 (4) , 1-26
Abstract
The learning objective plays a fundamental role to build a recommender system. Most methods routinely adopt either pointwise (e.g., binary cross-entropy) or pairwise (e.g., BPR) loss to train the model parameters, while rarely pay attention to softmax loss, which assumes the probabilities of all classes sum up to 1, due to its computational complexity when scaling up to large datasets or intractability for streaming data where the complete item space is not always available. The sampled softmax (SSM) loss emerges as an efficient substitute for softmax loss. Its special case, InfoNCE loss, has been widely used in self-supervised learning and exhibited remarkable performance for contrastive learning. Nonetheless, limited recommendation work uses the SSM loss as the learning objective. Worse still, none of them explores its properties thoroughly and answers “Does SSM loss suit for item recommendation?” and “What are the conceptual advantages of SSM loss, as compared with the prevalent losses?”, to the best of our knowledge. In this work, we aim at offering a better understanding of SSM for item recommendation. Specifically, we first theoretically reveal three model-agnostic advantages: (1) mitigating popularity bias, which is beneficial to long-tail recommendation; (2) mining hard negative samples, which offers informative gradients to optimize model parameters; and (3) maximizing the ranking metric, which facilitates top- K performance. However, based on our empirical studies, we recognize that the default choice of cosine similarity function in SSM limits its ability in learning the magnitudes of representation vectors. As such, the combinations of SSM with the models that also fall short in adjusting magnitudes (e.g., matrix factorization) may result in poor representations. One step further, we provide mathematical proof that message passing schemes in graph convolution networks can adjust representation magnitude according to node degree, which naturally compensates for the shortcoming of SSM. Extensive experiments on four benchmark datasets justify our analyses, demonstrating the superiority of SSM for item recommendation. Our implementations are available in both TensorFlow 1 and PyTorch. 2Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (9227010114, 62302321, 62372399, 62121002)
- University Synergy Innovation Program of Anhui Province (GXXT-2022-040)
- SuperComputing Center of Hangzhou City University
This publication has 37 references indexed in Scilit:
- Deep Neural Networks for YouTube Recommendations2016
- Deep Crossing2016
- Improving pairwise learning for item recommendation from implicit feedback2014
- FISM2013
- Optimizing top-n collaborative filtering via dynamic negative item sampling2013
- Matrix Factorization Techniques for Recommender SystemsComputer, 2009
- Factorization meets the neighborhood2008
- Listwise approach to learning to rank2008
- Learning to rank2007
- Cours d'économie politique1964