P3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning
Authors:
Open Access
- 6 July 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1956-1962
Abstract
Compared to other language tasks, applying pre-trained language models (PLMs) for search ranking often requires more nuances and training signals. In this paper, we identify and study the two mismatches between pre-training and ranking fine-tuning: the training schema gap regarding the differences in training objectives and model architectures, and the task knowledge gap considering the discrepancy between the knowledge needed in ranking and that learned during pre-training. To mitigate these gaps, we propose Pre-trained, Prompt-learned and Pre-finetuned Neural Ranker (P3 Ranker). P3 Ranker leverages prompt-based learning to convert the ranking task into a pre-training like schema and uses pre-finetuning to initialize the model on intermediate supervised tasks. Experiments on MS MARCO and Robust04 show the superior performances of P3 Ranker in few-shot ranking. Analyses reveal that P3 Ranker is able to better accustom to the ranking task through prompt-based learning and retrieve necessary ranking-oriented knowledge gleaned in pre-finetuning, resulting in data-efficient PLM adaptation. Our code is available at https://github.com/NEUIR/P3Ranker.Keywords
All Related Versions
Funding Information
- Natural Science Foundation of China (62006129)
- Fundamental Research Funds for the Central Universities (N2216013)
- National Science and Technology Major Project (J2019-IV-0002-0069)
- College Scientific Research Project of Inner Mongolia (NJZY21263)
This publication has 23 references indexed in Scilit:
- Selective Weak Supervision for Neural Information Retrieval2020
- Deeper Text Understanding for IR with Contextual Neural Language Modeling2019
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling2019
- Latent Retrieval for Weakly Supervised Open Domain Question Answering2019
- MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension2019
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- Anserini2017
- End-to-End Neural Ad-hoc Ranking with Kernel Pooling2017
- Reading Wikipedia to Answer Open-Domain Questions2017
- A Markov random field model for term dependencies2005