Prior-Data Fitted Networks as Tabular Foundation Models for Ranking in Low-Data Settings

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Learning to rank (LTR) traditionally requires large-scale training data to generalize effectively. In low-data domains where expert annotation is scarce, the performance of LTR methods degrades sharply. Foundation models have alleviated similar data dependencies in other domains via in-context learning, but a foundation model for ranking with tabular features has not been explored yet. We propose prior-data fitted networks (PFNs) as a strong method for ranking in low-data settings. First, we demonstrate that PFNs, which are originally trained for classification, successfully outperform classification baselines on ranking data. Next, we evaluate PFNs as rankers, showing that they surpass state-of-the-art tuned baselines in low-data regimes. We introduce a novel sampling and inference scheme to obtain pairwise predictions from PFNs' native pointwise architecture, analogous to pairwise LTR. To address the limited context window of the transformers underlying PFNs, we propose a dynamic support set selection strategy for queries that scales PFNs beyond random subsampling. Our experimental results show that PFNs are an effective foundation model for ranking that provides significant gains when data is limited.
Funding Information
  • Dutch Research Council (NWO) (24.004.022)
  • Dutch Research Council (NWO) (NWA.1389.20.- 183)
  • Dutch Research Council (NWO) (KICH3.LTP.20.006)
  • Dutch Research Council (NWO) (VI.Veni.222.269)
  • European Union (101201510 (UNITE))