NICO++: Towards Better Benchmarking for Domain Generalization
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16036-16047
Abstract
Despite the remarkable performance that modern deep neural networks have achieved on independent and identically distributed (I.I.D.) data, they can crash under distribution shifts. Most current evaluation methods for do-main generalization (DG) adopt the leave-one-out strategy as a compromise on the limited number of domains. We propose a large-scale benchmark with extensive labeled domains named $NICO^{++}$ along with more rational evaluation methods for comprehensively evaluating DC algorithms. To evaluate DG datasets, we propose two metrics to quantify covariate shift and concept shift, respectively. Two novel generalization bounds from the perspective of data construction are proposed to prove that limited concept shift and significant covariate shift favor the evaluation capability for generalization. Through extensive experiments, $NlCO^{++}$ shows its superior evaluation capability compared with current DG datasets and its contribution in alleviating unfairness caused by the leak of oracle knowledge in model selection. The data and code for the benchmark based on $NICO^{++}$ are available at https://github.com/xxgege/NICO-plus.
Keywords
All Related Versions
This publication has 56 references indexed in Scilit:
- Deep Domain Generalization With Structured Low-Rank ConstraintIEEE Transactions on Image Processing, 2017
- Transfer Learning for Domain Adaptation in MRI: Application in Brain Lesion Segmentation2017
- Deep Hashing Network for Unsupervised Domain Adaptation2017
- Deep learning for healthcare: review, opportunities and challengesBriefings in Bioinformatics, 2017
- Scatter Component Analysis: A Unified Framework for Domain Adaptation and Domain GeneralizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Learning Attributes Equals Multi-Source Domain Generalization2016
- Visual domain adaptation via transfer feature learningKnowledge and Information Systems, 2016
- LSDT: Latent Sparse Domain Transfer Learning for Visual AdaptationIEEE Transactions on Image Processing, 2016
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- Unbiased Metric Learning: On the Utilization of Multiple Datasets and Web Images for Softening Bias2013