CTR-Sink: Attention Sink for Language Models in Click-Through Rate Prediction
Authors:
- Zixuan LiZLZixuan Li
- Binzong GengBGBinzong Geng
- Jing XiongJXJing Xiong
- Yong HeYHYong He
- Yuxuan HuYHYuxuan Hu
- Jian ChenJCJian Chen
- Dingwei ChenDCDingwei Chen
- Xiyu ChangXCXiyu Chang
- Liang ZhangLZLiang Zhang
- Linjian MoLMLinjian Mo
- Chengming LiCLChengming Li
- Chuan YuanCYChuan Yuan
Preprint
Abstract
Click-Through Rate (CTR) prediction, a core task in recommendation systems, estimates user click likelihood using historical behavioral data. Modeling user behavior sequences as text to leverage Language Models (LMs) for this task has gained traction, owing to LMs' strong semantic understanding and contextual modeling capabilities. However, a critical structural gap exists: user behavior sequences consist of discrete actions connected by semantically empty separators, differing fundamentally from the coherent natural language in LM pre-training. This mismatch causes semantic fragmentation, where LM attention scatters across irrelevant tokens instead of focusing on meaningful behavior boundaries and inter-behavior relationships, degrading prediction performance. To address this, we propose $\textit{CTR-Sink}$, a novel framework introducing behavior-level attention sinks tailored for recommendation scenarios. Inspired by attention sink theory, it constructs attention focus sinks and dynamically regulates attention aggregation via external information. Specifically, we insert sink tokens between consecutive behaviors, incorporating recommendation-specific signals such as temporal distance to serve as stable attention sinks. To enhance generality, we design a two-stage training strategy that explicitly guides LM attention toward sink tokens and a attention sink mechanism that amplifies inter-sink dependencies to better capture behavioral correlations. Experiments on one industrial dataset and two open-source datasets (MovieLens, Kuairec), alongside visualization results, validate the method's effectiveness across scenarios.
Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- CTRL: Connect Collaborative and Language Model for CTR PredictionACM Transactions on Recommender Systems, 2025
- Balancing Efficiency and Effectiveness: An LLM-Infused Approach for Optimized CTR Prediction2025
- Singleton CTR Prediction: Towards Single-User Modeling Paradigm with Large Language Models2025
- CoLLM: Integrating Collaborative Embeddings Into Large Language Models for RecommendationIEEE Transactions on Knowledge and Data Engineering, 2025
- How Can Recommender Systems Benefit from Large Language Models: A SurveyACM Transactions on Information Systems, 2025
- FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction2024
- Breaking the Length Barrier: LLM-Enhanced CTR Prediction in Long Textual User Behaviors2024
- ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation2024
- BERT4CTR: An Efficient Framework to Combine Pre-trained Language Model with Non-textual Features for CTR Prediction2023
- Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction2020