Han-Jia Ye

The practice of knowledge distillation, wherein pre-trained models – termed ”teachers” – serve to enhance the performance of subsequent ”student” models, has seen a surge in popularity. With a lot of pre-trained models available, there is a pressing need to investigate effective mechanisms for utilizing these resources optimally. This paper tackles two main challenges inherent in developing such mechanisms. The initial challenge lies in the effective selection of the most advantageous models from a vast pool, given that exhaustively testing each pre-trained model is impractical. The second challenge stems from the diversity of pre-trained models, as it is common for a pre-trained model to have a different label space than that of the current task. This necessitates the development of a universal model reuse approach capable of integrating any pre-trained model, regardless of its unknown label set. To address these issues, we introduce a dual-phase framework named “Selective Cross-Label Distillation.” The first phase, termed model assessment, evaluates the semantic similarity between a potential pre-trained model and the target model through optimal transport. By determining the transportation cost, we pinpoint candidate models with lower costs to serve as our source models. The second phase, knowledge reuse, concentrates on minimizing the transportation cost between the chosen source models and the target model. This dual-phase approach enables efficient model selection and bridges the semantic gap between pre-trained models and the target task, leading to enhanced performance. The effectiveness of our framework in model selection and knowledge reuse is validated by experimental results.

Structured data constitutes one of the most ubiquitous data modalities in web-scale and enterprise applications, supporting tasks such as recommendation, forecasting, and user behavior analysis. Conventional modeling paradigms—ranging from generalized linear models and gradient boosting to deep structured networks—have provided strong baselines for predictive analytics and decision support. However, the recent emergence of foundation models and in-context learning (ICL) has sparked a new paradigm for structured modeling, moving from dataset-specific training toward universal, adaptable inference. Emerging structured foundation models illustrate how large-scale pretraining, synthetic data generation, and ICL-based inference can extend foundation-model principles to structured data. These developments open new possibilities for multi-task learning, zero-shot inference, and knowledge transfer across diverse structured settings. Yet, the space of structured foundation models remains largely unexplored, with open questions surrounding data generation, multi-task setting, pretraining objectives, and evaluation standards. This tutorial will provide a structured overview of both conventional modeling and recent ICL-based approaches. Participants will gain a comprehensive understanding of established methods, current advances in foundation models, and open research challenges. In particular, we will offer an in-depth introduction to structured ICL and review the most representative foundation models in this field. Several key topics in this field will be discussed, including pretraining data generation, multi-task learning, and other emerging directions in the modeling of structured data. This tutorial aims to bridge conventional machine learning and the emerging foundation-model paradigm, providing attendees with conceptual and practical insights into structured data modeling in the era of generalist foundation models.

LLM routing aims to achieve a favorable quality--cost trade-off by dynamically assigning easy queries to smaller models and harder queries to stronger ones. However, across both unimodal and multimodal settings, we uncover a pervasive yet underexplored failure mode in existing routers: as the user's cost budget increases, routers systematically default to the most capable and most expensive model even when cheaper models already suffice. As a result, current routers under-utilize small models, wasting computation and monetary cost and undermining the core promise of routing; we term this phenomenon routing collapse. We attribute routing collapse to an objective--decision mismatch: many routers are trained to predict scalar performance scores, whereas routing decisions ultimately depend on discrete comparisons among candidate models. Consequently, small prediction errors can flip relative orderings and trigger suboptimal selections. To bridge this gap, we propose EquiRouter, a decision-aware router that directly learns model rankings, restoring the role of smaller models and mitigating routing collapse. On RouterBench, EquiRouter reduces cost by about 17\% at GPT-4-level performance compared to the strongest prior router. Our code is available at https://github.com/AIGNLAI/EquiRouter.

Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications. Models for learning from tabular data have continuously evolved, with Deep Neural Networks (DNNs) recently demonstrating promising results through their capability of representation learning. In this survey, we systematically introduce the field of tabular representation learning, covering the background, challenges, and benchmarks, along with the pros and cons of using DNNs. We organize existing methods into three main categories according to their generalization capabilities: specialized, transferable, and general models. Specialized models focus on tasks where training and evaluation occur within the same data distribution. We introduce a hierarchical taxonomy for specialized models based on the key aspects of tabular data—features, samples, and objectives—and delve into detailed strategies for obtaining high-quality feature- and sample-level representations. Transferable models are pre-trained on one or more datasets and subsequently fine-tuned on downstream tasks, leveraging knowledge acquired from homogeneous or heterogeneous sources, or even cross-modalities such as vision and language. General models, also known as tabular foundation models, extend this concept further, allowing direct application to downstream tasks without additional fine-tuning. We group these general models based on the strategies used to adapt across heterogeneous datasets. Additionally, we explore ensemble methods, which integrate the strengths of multiple tabular models. Finally, we discuss representative extensions of tabular learning, including open-environment tabular machine learning, multimodal learning with tabular data, and tabular understanding tasks.