Improving BERT-based Query-by-Document Retrieval with Multi-Task Optimization
Preprint
Abstract
Query-by-document (QBD) retrieval is an Information Retrieval task in which a seed document acts as the query and the goal is to retrieve related documents -- it is particular common in professional search tasks. In this work we improve the retrieval effectiveness of the BERT re-ranker, proposing an extension to its fine-tuning step to better exploit the context of queries. To this end, we use an additional document-level representation learning objective besides the ranking objective when fine-tuning the BERT re-ranker. Our experiments on two QBD retrieval benchmarks show that the proposed multi-task optimization significantly improves the ranking effectiveness without changing the BERT re-ranker or using additional training samples. In future work, the generalizability of our approach to other retrieval tasks should be further investigated.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- COLIEE 2020: Methods for Legal Document Retrieval and Entailment2021
- LEGAL-BERT: The Muppets straight out of Law School2020
- SPECTER: Document-level Representation Learning using Citation-informed Transformers2020
- First International Workshop on Professional SearchACM SIGIR Forum, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- SciBERT: A Pretrained Language Model for Scientific Text2019
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks2019
- Multi-Task Deep Neural Networks for Natural Language Understanding2019
- Information retrieval in the workplace: A comparison of professional search practicesInformation Processing & Management, 2018
- Learning to rank using gradient descent2005