ALT: Towards Fine-grained Alignment between Language and CTR Models for Click-Through Rate Prediction
Preprint
Abstract
Click-through rate (CTR) prediction plays as a core function module in various personalized online services. According to the data modality and input format, the models for CTR prediction can be mainly classified into two categories. The first one is the traditional CTR models that take as inputs the one-hot encoded ID features of tabular modality, which aims to capture the collaborative signals via feature interaction modeling. The second category takes as inputs the sentences of textual modality obtained by hard prompt templates, where pretrained language models (PLMs) are adopted to extract the semantic knowledge. These two lines of research generally focus on different characteristics of the same input data (i.e., textual and tabular modalities), forming a distinct complementary relationship with each other. Therefore, in this paper, we propose to conduct fine-grained feature-level Alignment between Language and CTR models (ALT) for CTR prediction. Apart from the common CLIP-like instance-level contrastive learning, we further design a novel joint reconstruction pretraining task for both masked language and tabular modeling. Specifically, the masked data of one modality (i.e., tokens or features) has to be recovered with the help of the other modality, which establishes the feature-level interaction and alignment via sufficient mutual information extraction between dual modalities. Moreover, we propose three different finetuning strategies with the option to train the aligned language and CTR models separately or jointly for downstream CTR prediction tasks, thus accommodating the varying efficacy and efficiency requirements for industrial applications. Extensive experiments on three real-world datasets demonstrate that ALT outperforms SOTA baselines, and is highly compatible for various language and CTR models.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- How Can Recommender Systems Benefit from Large Language Models: A SurveyACM Transactions on Information Systems, 2025
- Towards Open-World Recommendation with Knowledge Augmentation from Large Language Models2024
- ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation2024
- ClickPrompt: CTR Models are Strong Prompt Generators for Adapting Language Models to CTR Prediction2024
- Pre-train, Prompt, and Recommendation: A Comprehensive Survey of Language Modeling Paradigm Adaptations in Recommender SystemsTransactions of the Association for Computational Linguistics, 2023
- VIP5: Towards Multimodal Foundation Models for Recommendation2023
- Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!2023
- TinyBERT: Distilling BERT for Natural Language Understanding2020
- Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction2019
- Practical Lessons from Predicting Clicks on Ads at Facebook2014