Fine-Grained Distillation for Long Document Retrieval
Preprint
Abstract
Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Multi-granularity cross-modal representation learning for named entity recognition on social mediaInformation Processing & Management, 2023
- Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction2022
- Vision-Language Pre-Training with Triple Contrastive Learning2022
- Jointly Optimizing Query Encoder and Product Quantization to Improve Retrieval Performance2021
- Few-Shot Conversational Dense Retrieval2021
- Optimizing Dense Retrieval Model Training with Hard Negatives2021
- B-PROP2021
- Rethink Training of BERT Rerankers in Multi-stage Retrieval Pipeline2021
- Complement Lexical Retrieval Model with Semantic Residual Embeddings2021
- Embedding-based Retrieval in Facebook Search2020