Sparse Coding with Multi-Layer Decoders using Variance Regularization
Authors:
Preprint
Abstract
Sparse representations of images are useful in many computer vision applications. Sparse coding with an $l_1$ penalty and a learned linear dictionary requires regularization of the dictionary to prevent a collapse in the $l_1$ norms of the codes. Typically, this regularization entails bounding the Euclidean norms of the dictionary's elements. In this work, we propose a novel sparse coding protocol which prevents a collapse in the codes without the need to regularize the decoder. Our method regularizes the codes directly so that each latent code component has variance greater than a fixed threshold over a set of sparse representations for a given set of inputs. Furthermore, we explore ways to effectively train sparse coding systems with multi-layer decoders since they can model more complex relationships than linear dictionaries. In our experiments with MNIST and natural image patches, we show that decoders learned with our approach have interpretable features both in the linear and multi-layer case. Moreover, we show that sparse autoencoders with multi-layer decoders trained using our variance regularization method produce higher quality reconstructions with sparser representations when compared to autoencoders with linear dictionaries. Additionally, sparse representations obtained with our variance regularization approach are useful in the downstream tasks of denoising and classification in the low-data regime.
Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Natural images are reliably represented by sparse and variable populations of neurons in visual cortexNature Communications, 2020
- Deep Dictionary Learning: A PARametric NETwork ApproachIEEE Transactions on Image Processing, 2019
- Nonlinear dictionary learning with application to image classificationPattern Recognition, 2018
- A Survey of Sparse Representation: Algorithms and ApplicationsIEEE Access, 2015
- Are sparse representations really relevant for image classification?2011
- What is the best multi-stage architecture for object recognition?2009
- Sparse Representation for Color Image RestorationIEEE Transactions on Image Processing, 2007
- Image Denoising Via Sparse and Redundant Representations Over Learned DictionariesIEEE Transactions on Image Processing, 2006
- Forming sparse representations by local anti-Hebbian learningBiological Cybernetics, 1990