Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation
Authors:
Top Cited Papers
Open Access
- 11 February 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 813-823
Abstract
Recent advancements of sequential deep learning models such as Transformer and BERT have significantly facilitated the sequential recommendation. However, according to our study, the distribution of item embeddings generated by these models tends to degenerate into an anisotropic shape, which may result in high semantic similarities among embeddings. In this paper, both empirical and theoretical investigations of this representation degeneration problem are first provided, based on which a novel recommender model DuoRec is proposed to improve the item embeddings distribution. Specifically, in light of the uniformity property of contrastive learning, a contrastive regularization is designed for DuoRec to reshape the distribution of sequence representations. Given the convention that the recommendation task is performed by measuring the similarity between sequence representations and item embeddings in the same space via dot product, the regularization can be implicitly applied to the item embedding distribution. Existing contrastive learning methods mainly rely on data level augmentation for user-item interaction sequences through item cropping, masking, or reordering and can hardly provide semantically consistent augmentation samples. In DuoRec, a model-level augmentation is proposed based on Dropout to enable better semantic preserving. Furthermore, a novel sampling strategy is developed, where sequences having the same target item are chosen hard positive samples. Extensive experiments conducted on five datasets demonstrate the superior performance of the proposed DuoRec model compared with baseline methods. Visualization results of the learned representations validate that DuoRec can largely alleviate the representation degeneration problem.Keywords
All Related Versions
Funding Information
- Australian Research Council (CE200100025,DP190102353,DP190101985,FT210100624)
This publication has 10 references indexed in Scilit:
- Self-Supervised Hypergraph Convolutional Networks for Session-based RecommendationProceedings of the AAAI Conference on Artificial Intelligence, 2021
- On Sampled Metrics for Item Recommendation2020
- Exploiting Cross-session Information for Session-based Recommendation with Graph Neural NetworksACM Transactions on Information Systems, 2020
- BERT4Rec2019
- How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings2019
- Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding2018
- Discrete Deep Learning for Fast Content-Aware Recommendation2018
- Image-Based Recommendations on Styles and Substitutes2015
- Factorizing personalized Markov chains for next-basket recommendation2010
- Item-based collaborative filtering recommendation algorithms2001