Large Language Models as Attribution Regularizers for Efficient Model Training
Open Access
- 12 October 2025
- book chapter
- Published by Springer Nature
- p. 68-90
Abstract
Large Language Models (LLMs) have demonstrated remarkable performance across diverse domains. However, effectively leveraging their vast knowledge for training smaller downstream models remains an open challenge, especially in domains like tabular data learning, where simpler models are often preferred due to interpretability and efficiency. In this paper, we introduce a novel yet straightforward method for incorporating LLM-generated global task feature attributions into the training process of smaller networks. Specifically, we propose an attribution-matching regularization term that aligns the training dynamics of the smaller model with the insights provided by the LLM. By doing so, our approach yields superior performance in few-shot learning scenarios. Notably, our method requires only black-box API access to the LLM, making it easy to integrate into existing training pipelines with minimal computational overhead. Furthermore, we demonstrate how this method can be used to address common issues in real-world datasets, such as skewness and bias. By integrating high-level knowledge from LLMs, our approach improves generalization, even when training data is limited or imbalanced. We validate its effectiveness through extensive experiments across multiple tasks, demonstrating improved learning efficiency and model robustness.Keywords
All Related Versions
This publication has 35 references indexed in Scilit:
- Accurate predictions on small data with a tabular foundation modelNature, 2025
- Going Beyond XAI: A Systematic Survey for Explanation-Guided LearningACM Computing Surveys, 2024
- Transparency of artificial intelligence/machine learning-enabled medical devicesnpj Digital Medicine, 2024
- Ethics and discrimination in artificial intelligence-enabled recruitment practicesHumanities and Social Sciences Communications, 2023
- RES: A Robust Framework for Guiding Visual Explanation2022
- Tabular data: Deep learning is not all you needInformation Fusion, 2022
- Improving performance of deep learning models with axiomatic attribution priors and expected gradientsNature Machine Intelligence, 2021
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based LocalizationInternational Journal of Computer Vision, 2019
- XGBoost2016
- GENERALIZED BODY COMPOSITION PREDICTION EQUATION FOR MEN USING SIMPLE MEASUREMENT TECHNIQUESMedicine & Science in Sports & Exercise, 1985