Calibration-compatible Listwise Distillation of Privileged Features for CTR Prediction
Open Access
- 4 March 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 247-256
Abstract
In machine learning systems, privileged features refer to the features that are available during offline training but inaccessible for online serving. Previous studies have recognized the importance of privileged features and explored ways to tackle online-offline discrepancies. A typical practice is privileged features distillation (PFD): train a teacher model using all features (including privileged ones) and then distill the knowledge from the teacher model using a student model (excluding the privileged features), which is then employed for online serving. In practice, the pointwise cross-entropy loss is often adopted for PFD. However, this loss is insufficient to distill the ranking ability for CTR prediction. First, it does not consider the non-i.i.d. characteristic of the data distribution, i.e., other items on the same page significantly impact the click probability of the candidate item. Second, it fails to consider the relative item order ranked by the teacher model's predictions, which is essential to distill the ranking ability. To address these issues, we first extend the pointwise-based PFD to the listwise-based PFD. We then define the calibration-compatible property of distillation loss and show that commonly used listwise losses do not satisfy this property when employed as distillation loss, thus compromising the model's calibration ability, which is another important measure for CTR prediction. To tackle this dilemma, we propose Calibration-compatible LIstwise Distillation (CLID), which employs carefully-designed listwise distillation loss to achieve better ranking ability than the pointwise-based PFD while preserving the model's calibration ability. We theoretically prove it is calibration-compatible. Extensive experiments on public datasets and a production dataset collected from the display advertising system of Alibaba further demonstrate the effectiveness of CLID.Keywords
All Related Versions
Funding Information
- Taishan Scholar Program of Shandong Province in China
- National Natural Science Foundation of China (No. 62072380 and 62272276)
- National Key Research and Development Program of China (2022YFC3502101)
- Alibaba Innovation Research Program
This publication has 45 references indexed in Scilit:
- Learning with Privileged Information for Efficient Image Super-Resolution2020
- Learning to Rank with Selection Bias in Personal Search2016
- Emotion Recognition with the Help of Privileged InformationIEEE Transactions on Autonomous Mental Development, 2015
- Reusing historical interaction data for faster online learning to rank for IR2013
- A general approximation framework for direct optimization of information retrieval measuresInformation Retrieval Journal, 2009
- Statistical Analysis of Bayes Optimal Subset RankingIEEE Transactions on Information Theory, 2008
- Query-level loss functions for information retrievalInformation Processing & Management, 2008
- Listwise approach to learning to rank2008
- Learning to rank2007
- Learning to rank using gradient descent2005