The Geometry of Feature Space in Deep Learning Models: A Holistic Perspective and Comprehensive Review
Author:
Open Access
- 19 May 2023
- journal article
- review article
- Published by MDPI AG in Mathematics
- Vol. 11 (10) , 2375
Abstract
As the field of deep learning experiences a meteoric rise, the urgency to decipher the complex geometric properties of feature spaces, which underlie the effectiveness of diverse learning algorithms and optimization techniques, has become paramount. In this scholarly review, a comprehensive, holistic outlook on the geometry of feature spaces in deep learning models is provided in order to thoroughly probe the interconnections between feature spaces and a multitude of influential factors such as activation functions, normalization methods, and model architectures. The exploration commences with an all-encompassing examination of deep learning models, followed by a rigorous dissection of feature space geometry, delving into manifold structures, curvature, wide neural networks and Gaussian processes, critical points and loss landscapes, singular value spectra, and adversarial robustness, among other notable topics. Moreover, transfer learning and disentangled representations in feature space are illuminated, accentuating the progress and challenges in these areas. In conclusion, the challenges and future research directions in the domain of feature space geometry are outlined, emphasizing the significance of comprehending overparameterized models, unsupervised and semi-supervised learning, interpretable feature space geometry, topological analysis, and multimodal and multi-task learning. Embracing a holistic perspective, this review aspires to serve as an exhaustive guide for researchers and practitioners alike, clarifying the intricacies of the geometry of feature spaces in deep learning models and mapping the trajectory for future advancements in this enigmatic and enthralling domain.Keywords
Funding Information
- Generative Artificial Intelligence System Inc. (GAIS)
This publication has 64 references indexed in Scilit:
- Topological Data AnalysisAnnual Review of Statistics and Its Application, 2018
- Multimodal Multi-task Learning for Dimensional and Continuous Emotion Recognition2017
- Analysis of classifiers’ robustness to adversarial perturbationsMachine Learning, 2017
- Harmonic Networks: Deep Translation and Rotation Equivariance2017
- Improving Training of Deep Neural Networks via Singular Value Bounding2017
- Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs2017
- Why Deep Learning Works: A Manifold Disentanglement PerspectiveIEEE Transactions on Neural Networks and Learning Systems, 2015
- Deep learningNature, 2015
- Scene Recognition by Manifold Regularized Deep Learning ArchitectureIEEE Transactions on Neural Networks and Learning Systems, 2015
- Persistent homology—a survey2008