HORAE: Temporal Multi-Interest Pre-training for Sequential Recommendation
Open Access
- 9 May 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Information Systems
- Vol. 43 (4) , 1-29
Abstract
The data sparsity problem has been a long-standing obstacle towards achieving better recommendation performance since it is miserable to estimate the user’s interests from limited historical behaviors. The pre-training paradigm, i.e., learning universal knowledge across a wide spectrum of domains, has increasingly become a new de-facto practice in many fields, especially for adaption to new domains. The merit of this superior generalizability renders it a natural choice to tackle the data sparsity problem for various recommendation scenarios. Hence, several efforts mainly follow masked language modeling or simple data augmentation via contrastive learning to build a pre-trained recommendation model. Our recent work (namely Miracle ) suggests that the common treatment utilizing the masked language modeling is not sufficient for pre-training a recommender system, since a user’s intent could be more complex than predicting the next word or item. The encouraging results demonstrate that the multi-interest modeling could significantly push the frontier of recommender system pre-training. Nevertheless, how to accommodate the temporal dynamics of the user interests seems to be underexplored under both single vector representation and multi-interest schemes. In this article, we aim to incorporate sophisticated temporal information modeling with the current advance in this line. More specifically, we extend Miracle by further considering relative position information and two kinds of relative time interval information jointly when performing multi-interest learning. Then, a sequential process for interest refinement is proposed to learn the subtle nuances of how interests change and shift along the timeline, leading to a more precise representation of user interests. Our extensive experiments on multiple real-world datasets validate the effectiveness of the proposed solution, demonstrating a significant improvement over current state-of-the-art models on these benchmarks. The code is available at https://github.com/WHUIR/Horae .Keywords
This publication has 14 references indexed in Scilit:
- User-Aware Multi-Interest Learning for Candidate Matching in Recommenders2022
- When Multi-Level Meets Multi-Interest2022
- Exploiting Positional Information for Session-Based RecommendationACM Transactions on Information Systems, 2021
- The World is Binary: Contrastive Learning for Denoising Next Basket Recommendation2021
- S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization2020
- Controllable Multi-Interest Framework for Recommendation2020
- BERT4Rec2019
- Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks2019
- Self-Attentive Sequential Recommendation2018
- Factorizing personalized Markov chains for next-basket recommendation2010