Towards Reliable Domain Generalization: A New Dataset and Evaluations
Preprint
Abstract
There are ubiquitous distribution shifts in the real world. However, deep neural networks (DNNs) are easily biased towards the training set, which causes severe performance degradation when they receive out-of-distribution data. Many methods are studied to train models that generalize under various distribution shifts in the literature of domain generalization (DG). However, the recent DomainBed and WILDS benchmarks challenged the effectiveness of these methods. Aiming at the problems in the existing research, we propose a new domain generalization task for handwritten Chinese character recognition (HCCR) to enrich the application scenarios of DG method research. We evaluate eighteen DG methods on the proposed PaHCC (Printed and Handwritten Chinese Characters) dataset and show that the performance of existing methods on this dataset is still unsatisfactory. Besides, under a designed dynamic DG setting, we reveal more properties of DG methods and argue that only the leave-one-domain-out protocol is unreliable. We advocate that researchers in the DG community refer to dynamic performance of methods for more comprehensive and reliable evaluation. Our dataset and evaluations bring new perspectives to the community for more substantial progress. We will make our dataset public with the article published to facilitate the study of domain generalization.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Towards Non-I.I.D. image classification: A dataset and baselinesPattern Recognition, 2021
- Learning to Optimize Domain Specific Normalization for Domain Generalization2020
- Self-challenging Improves Cross-Domain Generalization2020
- Deep Domain Generalization via Conditional Invariant Adversarial Networks2018
- Domain Generalization with Adversarial Feature Learning2018
- Deep Hashing Network for Unsupervised Domain Adaptation2017
- Online and offline handwritten Chinese character recognition: Benchmarking on new databasesPattern Recognition, 2013
- Unbiased look at dataset bias2011
- Style transfer matrix learning for writer adaptation2011
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998