On Generalization Bounds for Deep Compound Gaussian Neural Networks
Preprint
Abstract
Algorithm unfolding or unrolling is the technique of constructing a deep neural network (DNN) from an iterative algorithm. Unrolled DNNs often provide better interpretability and superior empirical performance over standard DNNs in signal estimation tasks. An important theoretical question, which has only recently received attention, is the development of generalization error bounds for unrolled DNNs. These bounds deliver theoretical and practical insights into the performance of a DNN on empirical datasets that are distinct from, but sampled from, the probability density generating the DNN training data. In this paper, we develop novel generalization error bounds for a class of unrolled DNNs that are informed by a compound Gaussian prior. These compound Gaussian networks have been shown to outperform comparative standard and unfolded deep neural networks in compressive sensing and tomographic imaging problems. The generalization error bound is formulated by bounding the Rademacher complexity of the class of compound Gaussian network estimates with Dudley's integral. Under realistic conditions, we show that, at worst, the generalization error scales $\mathcal{O}(n\sqrt{\ln(n)})$ in the signal dimension and $\mathcal{O}(($Network Size$)^{3/2})$ in network size.
Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Deep Regularized Compound Gaussian Network for Solving Linear Inverse ProblemsIEEE Transactions on Computational Imaging, 2024
- Deep Memory-Augmented Proximal Unrolling Network for Compressive SensingInternational Journal of Computer Vision, 2023
- Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image ProcessingIEEE Signal Processing Magazine, 2021
- Waveform Optimization for Multistatic Radar Imaging Using Mutual InformationIEEE Transactions on Aerospace and Electronic Systems, 2021
- Loss Functions for Image Restoration With Neural NetworksIEEE Transactions on Computational Imaging, 2016
- A hierarchical Bayesian-MAP approach to inverse problems in imagingInverse Problems, 2016
- A Mathematical Introduction to Compressive Sensing2013
- Random Cascades on Wavelet Trees and Their Use in Analyzing and Modeling Natural ImagesApplied and Computational Harmonic Analysis, 2001
- Importance of input data normalization for the application of neural networks to complex industrial problemsIEEE Transactions on Nuclear Science, 1997
- A Probabilistic Theory of Pattern Recognition1996