Training Curricula for Open Domain Answer Re-Ranking
Authors:
- Sean MacAvaneySMSean MacAvaney
- Franco Maria NardiniFNFranco Maria Nardini
- Raffaele PeregoRPRaffaele Perego
- Nicola TonellottoNTNicola Tonellotto
- Nazli GoharianNGNazli Goharian
- Ophir FriederOFOphir Frieder
Preprint
Abstract
In precision-oriented tasks like answer ranking, it is more important to rank many relevant answers highly than to retrieve all relevant answers. It follows that a good ranking strategy would be to learn how to identify the easiest correct answers first (i.e., assign a high ranking score to answers that have characteristics that usually indicate relevance, and a low ranking score to those with characteristics that do not), before incorporating more complex logic to handle difficult cases (e.g., semantic matching or reasoning). In this work, we apply this idea to the training of neural answer rankers using curriculum learning. We propose several heuristics to estimate the difficulty of a given training sample. We show that the proposed heuristics can be used to build a training curriculum that down-weights difficult samples early in the training process. As the training process progresses, our approach gradually shifts to weighting all samples equally, regardless of difficulty. We present a comprehensive evaluation of our proposed idea on three answer ranking datasets. Results show that our approach leads to superior performance of two leading neural ranking architectures, namely BERT and ConvKNRM, using both pointwise and pairwise losses. When applied to a BERT-based ranker, our method yields up to a 4% improvement in MRR and a 9% improvement in P@1 (compared to the model trained without a curriculum). This results in models that can achieve comparable performance to more expensive state-of-the-art techniques.Keywords
All Related Versions
- Version 1, 2020-04-29, ArXiv
- Version 2, 2020-05-21, ArXiv
- Published version: , 529.
This publication has 16 references indexed in Scilit:
- ANTIQUE: A Non-factoid Question Answering Benchmark2020
- Efficient Interaction-based Neural Ranking with Locality Sensitive Hashing2019
- Overcoming low-utility facets for complex answer retrievalInformation Retrieval Journal, 2018
- Continuation Methods and Curriculum Learning for Learning to Rank2018
- Curriculum Learning for Heterogeneous Star Network Embedding via Deep Reinforcement Learning2018
- Convolutional Neural Networks for Soft-Matching N-Grams in Ad-hoc Search2018
- Learning semantic representations using convolutional neural networks for web search2014
- Learning deep structured semantic models for web search using clickthrough data2013
- A Markov random field model for term dependencies2005
- Parallel continuation-based global optimization for molecular conformation and protein foldingJournal of Global Optimization, 1996