Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models
Authors:
Preprint
Abstract
Fine-tuning foundation models often compromises their robustness to distribution shifts. To remedy this, most robust fine-tuning methods aim to preserve the pre-trained features. However, not all pre-trained features are robust and those methods are largely indifferent to which ones to preserve. We propose dual risk minimization (DRM), which combines empirical risk minimization with worst-case risk minimization, to better preserve the core features of downstream tasks. In particular, we utilize core-feature descriptions generated by LLMs to induce core-based zero-shot predictions which then serve as proxies to estimate the worst-case risk. DRM balances two crucial aspects of model robustness: expected performance and worst-case performance, establishing a new state of the art on various real-world benchmarks. DRM significantly improves the out-of-distribution performance of CLIP ViT-L/14@336 on ImageNet (75.9 to 77.1), WILDS-iWildCam (47.1 to 51.8), and WILDS-FMoW (50.7 to 53.1); opening up new avenues for robust fine-tuning. Our code is available at https://github.com/vaynexie/DRM .Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Context-Aware Robust Fine-TuningInternational Journal of Computer Vision, 2023
- CLIP-Adapter: Better Vision-Language Models with Feature AdaptersInternational Journal of Computer Vision, 2023
- Combined scaling for zero-shot transfer learningNeurocomputing, 2023
- Domain Generalization by Mutual-Information Regularization with Pre-trained Models2022
- Causality Inspired Representation Learning for Domain Generalization2022
- Robust fine-tuning of zero-shot models2022
- Natural Adversarial Examples2021
- SpotTune: Transfer Learning Through Adaptive Fine-Tuning2019
- Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-Tuning2017
- Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem1989