QUERT: Continual Pre-training of Language Model for Query Understanding in Travel Domain Search
Open Access
- 4 August 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 5282-5291
Abstract
In light of the success of the pre-trained language models (PLMs), continual pre-training of generic PLMs has been the paradigm of domain adaption. In this paper, we propose QUERT, A Continual Pre-trained Language Model for QUERy Understanding in Travel Domain Search. QUERT is jointly trained on four tailored pre-training tasks to the characteristics of query in travel domain search: Geography-aware Mask Prediction, Geohash Code Prediction, User Click Behavior Learning, and Phrase and Token Order Prediction. Performance improvement of downstream tasks and ablation experiment demonstrate the effectiveness of our proposed pre-training tasks. To be specific, the average performance of downstream tasks increases by 2.02% and 30.93% in supervised and unsupervised settings, respectively. To check on the improvement of QUERT to online business, we deploy QUERT and perform A/B testing on Fliggy APP. The feedback results show that QUERT increases the Unique Click-Through Rate and Page Click-Through Rate by 0.89% and 1.03% when applying QUERT as the encoder. Resources are available at https://github.com/hsaest/QUERTKeywords
All Related Versions
Funding Information
- Science and Technology Commission of Shanghai Municipality (No. 23YF1409400, No. 22511105902, No.2021SHZDZX0103)
This publication has 17 references indexed in Scilit:
- Pre-Training With Whole Word Masking for Chinese BERTIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval2021
- SpanBERT: Improving Pre-training by Representing and Predicting SpansTransactions of the Association for Computational Linguistics, 2020
- BOND: BERT-Assisted Open-Domain Named Entity Recognition with Distant Supervision2020
- BERTweet: A pre-trained language model for English Tweets2020
- Transformers: State-of-the-Art Natural Language Processing2020
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension2020
- Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- ERNIE: Enhanced Language Representation with Informative Entities2019