Regularizing Neural Networks via Minimizing Hyperspherical Energy
Preprint
Abstract
Inspired by the Thomson problem in physics where the distribution of multiple propelling electrons on a unit sphere can be modeled via minimizing some potential energy, hyperspherical energy minimization has demonstrated its potential in regularizing neural networks and improving their generalization power. In this paper, we first study the important role that hyperspherical energy plays in neural network training by analyzing its training dynamics. Then we show that naively minimizing hyperspherical energy suffers from some difficulties due to highly non-linear and non-convex optimization as the space dimensionality becomes higher, therefore limiting the potential to further improve the generalization. To address these problems, we propose the compressive minimum hyperspherical energy (CoMHE) as a more effective regularization for neural networks. Specifically, CoMHE utilizes projection mappings to reduce the dimensionality of neurons and minimizes their hyperspherical energy. According to different designs for the projection mapping, we propose several distinct yet well-performing variants and provide some theoretical guarantees to justify their effectiveness. Our experiments show that CoMHE consistently outperforms existing regularization methods, and can be easily applied to different neural networks.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Identity Mappings in Deep Residual Networks2016
- Estimating the Number of Stable Configurations for the Generalized Thomson ProblemJournal of Statistical Physics, 2015
- Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation2014
- One‐Bit Compressed Sensing by Linear ProgrammingCommunications on Pure and Applied Mathematics, 2013
- Classification and clustering via dictionary learning with structured incoherence and shared features2010
- A Simple Proof of the Restricted Isometry Property for Random MatricesConstructive Approximation, 2008
- A Sharp Form of the Cramér–Wold TheoremJournal of Theoretical Probability, 2007
- An elementary proof of a theorem of Johnson and LindenstraussRandom Structures & Algorithms, 2002
- Finding Frequent Items in Data Streams2002
- Efficiency of generalized simulated annealingPhysical Review E, 2000