Blend and Match: Distilling Semantic Search Models with Different Inductive Biases and Model Architectures
Authors:
- Hamed BonabHBHamed Bonab
- Ashutosh JoshiAJAshutosh Joshi
- Ravi BhatiaRBRavi Bhatia
- Ankit GandhiAGAnkit Gandhi
- Vijay HuddarVHVijay Huddar
- Juhi NaikJNJuhi Naik
- Mutasem Al-DarabsahMAMutasem Al-Darabsah
- Choon Hui TeoCTChoon Hui Teo
- Jonathan MayJMJonathan May
- Tarun AgarwalTATarun Agarwal
- Vaclav PetricekVPVaclav Petricek
Open Access
- 30 April 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 869-877
Abstract
Commercial search engines use different semantic models to augment lexical matches. These models provide candidate items for a user’s query from a target space of millions to billions of items. Models with different inductive biases provide relatively different predictions, making it desirable to launch multiple semantic models in production. However, latency and resource constraints make simultaneously deploying multiple models impractical. In this paper, we introduce a distillation approach, called Blend and Match (BM), to unify two different semantic search models into a single model. We use a Bi-encoder semantic matching model as our primary model and propose a novel loss function to incorporate eXtreme Multi-label Classification (XMC) predictions as the secondary model. Our experiments conducted on two large-scale datasets, collected from a popular e-commerce store, show that our proposed approach significantly improves the recall of the primary Bi-encoder model by 11% to 17% with a minimal loss in precision. We show that traditional knowledge distillation approaches result in a sub-optimal performance for our problem setting, and our BM approach yields comparable rankings with strong Rank Fusion (RF) methods used only if one could deploy multiple models.Keywords
This publication has 24 references indexed in Scilit:
- Ranking Distillation2018
- Universal Approximation Functions for Fast Learning to Rank2018
- Retrieval Consistency in the Presence of Query Variations2017
- Learning from Multiple Teacher Networks2017
- Learning deep structured semantic models for web search using clickthrough data2013
- Reciprocal rank fusion outperforms condorcet and individual rank learning methods2009
- Learning to rank for information retrieval (LR4IR 2007)ACM SIGIR Forum, 2007
- Model compression2006
- ProbFuse2006
- A vector space model for automatic indexingCommunications of the ACM, 1975