Measuring Disentanglement: A Review of Metrics
- 14 November 2022
- journal article
- review article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 35 (7) , 8747-8761
Abstract
Learning to disentangle and represent factors of variation in data is an important problem in artificial intelligence. While many advances have been made to learn these representations, it is still unclear how to quantify disentanglement. While several metrics exist, little is known on their implicit assumptions, what they truly measure, and their limits. In consequence, it is difficult to interpret results when comparing different representations. In this work, we survey supervised disentanglement metrics and thoroughly analyze them. We propose a new taxonomy in which all metrics fall into one of the three families: intervention-based, predictor-based, and information-based. We conduct extensive experiments in which we isolate properties of disentangled representations, allowing stratified comparison along several axes. From our experiment results and analysis, we provide insights on relations between disentangled representation properties. Finally, we share guidelines on how to measure disentanglement.Keywords
All Related Versions
Funding Information
- Mitacs in Canada
- Ubisoft Montreal
This publication has 10 references indexed in Scilit:
- Shortcut learning in deep neural networksNature Machine Intelligence, 2020
- A Study of Inductive Biases for Unsupervised Speech Representation LearningIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020
- Variational Autoencoders Pursue PCA Directions (by Accident)2019
- Multi-Level Variational Autoencoder: Learning Disentangled Representations From Grouped ObservationsProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Learning methods for generic object recognition with invariance to pose and lighting2004
- Estimation of Entropy and Mutual InformationNeural Computation, 2003
- Random ForestsMachine Learning, 2001
- Learning Factorial Codes by Predictability MinimizationNeural Computation, 1992
- Rank Correlation Methods.Biometrika, 1957