Exploring Weight Distributions and Dependence in Neural Networks With $\alpha$-Stable Distributions
- 5 June 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 5 (11) , 5519-5529
Abstract
The fundamental use of neural networks is in providing a nonlinear mapping between input and output data with possibly a high number of parameters that can be learned from data directly. Consequently, studying the model’s parameters, particularly the weights, is of paramount importance. The distribution and interdependencies of these weights have a direct impact on the model’s generalizability, compressibility, initialization, and convergence speed. By fitting the weights of pretrained neural networks using the α-stable distributions and conducting statistical tests, we discover widespread heavy-tailed phenomena in neural network weights, with a few layers exhibiting asymmetry. Additionally, we employ a multivariate α-stable distribution to model the weights and explore the relationship between weights within and across layers by calculating the signed symmetric covariation coefficient. The results reveal a strong dependence among certain weights. Our findings indicate that the Gaussian assumption, symmetry assumption, and independence assumption commonly used in neural network research might be inconsistent with reality. In conclusion, our research shows three properties observed in neural network weights: heavy-tailed phenomena, asymmetry and dependence on certain weights.Keywords
Funding Information
- Shenzhen Science and Technology Innovation Commission (JCYJ20220530143002005)
- Shenzhen Ubiquitous Data Enabling Key Lab (ZDSYS20220527171406015)
- Tsinghua Shenzhen International Graduate School Startup fund (QD2022024C)
This publication has 29 references indexed in Scilit:
- Correlated Initialization for Correlated DataNeural Processing Letters, 2022
- Modelling impulsive noise in indoor powerline communication systemsSignal, Image and Video Processing, 2020
- Univariate Stable Distributions2020
- Trained Neural Networks Ensembles Weight Connections Analysis2018
- Mutual information based weight initialization method for sigmoidal feedforward neural networksNeurocomputing, 2016
- Bayesian data analysisWIREs Cognitive Science, 2010
- Signed symmetric covariation coefficient for alpha-stable dependence modelingComptes Rendus Mathematique, 2009
- Testing for Impulsive Behavior: A Bootstrap ApproachDigital Signal Processing, 2001
- Use of α-stable self-similar stochastic processes for modeling traffic in broadband networksPerformance Evaluation, 2000
- Do Backpropagation Trained Neural Networks have Normal Weight Distributions?1993