FLIP: Towards Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction
Preprint
Abstract
Click-through rate (CTR) prediction plays as a core function module in various personalized online services. The traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality, which capture the collaborative signals via feature interaction modeling. But the one-hot encoding discards the semantic information conceived in the original feature texts. Recently, the emergence of Pretrained Language Models (PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality obtained by hard prompt templates and adopts PLMs to extract the semantic knowledge. However, PLMs generally tokenize the input text data into subword tokens and ignore field-wise collaborative signals. Therefore, these two lines of research focus on different characteristics of the same input data (i.e., textual and tabular modalities), forming a distinct complementary relationship with each other. In this paper, we propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models (FLIP) for CTR prediction. We design a novel joint reconstruction pretraining task for both masked language and tabular modeling. Specifically, the masked data of one modality (i.e., tokens or features) has to be recovered with the help of the other modality, which establishes the feature-level interaction and alignment via sufficient mutual information extraction between dual modalities. Moreover, we propose to jointly finetune the ID-based model and PLM for downstream CTR prediction tasks, thus achieving superior performance by combining the advantages of both models. Extensive experiments on three real-world datasets demonstrate that FLIP outperforms SOTA baselines, and is highly compatible for various ID-based models and PLMs.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Bridging the Information Gap Between Domain-Specific Model and General LLM for Personalized Recommendation2024
- A survey on large language models for recommendationWorld Wide Web, 2024
- When large language models meet personalization: perspectives of challenges and opportunitiesWorld Wide Web, 2024
- Representation Learning with Large Language Models for Recommendation2024
- Recommender Systems in the Era of Large Language Models (LLMs)IEEE Transactions on Knowledge and Data Engineering, 2024
- Text Is All You Need: Learning Language Representations for Sequential Recommendation2023
- Pre-trained models for natural language processing: A surveyScience China Technological Sciences, 2020
- Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction2019
- Field-aware Factorization Machines for CTR Prediction2016
- Practical Lessons from Predicting Clicks on Ads at Facebook2014