Hangul Fonts Dataset: a Hierarchical and Compositional Dataset for Investigating Learned Representations
Preprint
Abstract
Hierarchy and compositionality are common latent properties in many natural and scientific datasets. Determining when a deep network's hidden activations represent hierarchy and compositionality is important both for understanding deep representation learning and for applying deep networks in domains where interpretability is crucial. However, current benchmark machine learning datasets either have little hierarchical or compositional structure, or the structure is not known. This gap impedes precise analysis of a network's representations and thus hinders development of new methods that can learn such properties. To address this gap, we developed a new benchmark dataset with known hierarchical and compositional structure. The Hangul Fonts Dataset (HFD) is comprised of 35 fonts from the Korean writing system (Hangul), each with 11,172 blocks (syllables) composed from the product of initial consonant, medial vowel, and final consonant glyphs. All blocks can be grouped into a few geometric types which induces a hierarchy across blocks. In addition, each block is composed of individual glyphs with rotations, translations, scalings, and naturalistic style variation across fonts. We find that both shallow and deep unsupervised methods only show modest evidence of hierarchy and compositionality in their representations of the HFD compared to supervised deep networks. Supervised deep network representations contain structure related to the geometrical hierarchy of the characters, but the compositional structure of the data is not evident. Thus, HFD enables the identification of shortcomings in existing methods, a critical first step toward developing new machine learning algorithms to extract hierarchical and compositional structure in the context of naturalistic variability.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Deep learning approaches for neural decoding across architectures and recording modalitiesBriefings in Bioinformatics, 2020
- Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortexPLOS Computational Biology, 2019
- FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery2019
- The Building Blocks of InterpretabilityDistill, 2018
- Emergence of Invariance and Disentanglement in Deep Representations2018
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding2018
- Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models2014
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Learning Factorial Codes by Predictability MinimizationNeural Computation, 1992