$L_p$-Norm Constrained Coding With Frank-Wolfe Network
Preprint
Abstract
We investigate the problem of $L_p$-norm constrained coding, i.e. converting signal into code that lies inside the $L_p$-ball and most faithfully reconstructs the signal. While previous works known as sparse coding have addressed the cases of $\ell_0$ "norm" and $L_1$-norm, more general cases with other $p$ values, especially with unknown $p$, remain a difficulty. We propose the Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an $L_p$-norm constrained problem. We show that the Frank-Wolfe solver for the $L_p$-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by $p$ and termed $pool_p$. The $pool_p$ unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep models either heuristically designed or converted from projection gradient descent or proximal algorithms. We further show that the hyper-parameter $p$ can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the $L_p$-norm constrained coding problem with unknown $p$. A convolutional extension of F-W Net is then presented. We evaluate the performance of F-W Net on an extensive range of simulations to show the strong learning capability of F-W Net. We then adopt F-W Net or Convolutional F-W Net on a series of real-data tasks that are all formulated as $L_p$-norm constrained coding, including image classification, image denoising, and super-resolution, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness. In particular, F-W Net achieves significantly better performance than the state-of-the-art convolutional networks on image denoising, leading to more than 2 dB gain on the BSD-68 dataset.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Learned Convolutional Sparse Coding2018
- Non-local Color Image Denoising with Convolutional Neural Networks2017
- Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image DenoisingIEEE Transactions on Image Processing, 2017
- Learning Optimal Nonlinearities for Iterative Thresholding AlgorithmsIEEE Signal Processing Letters, 2016
- PCANet: A Simple Deep Learning Baseline for Image Classification?IEEE Transactions on Image Processing, 2015
- Invariant Scattering Convolution NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- The Convex Geometry of Linear Inverse ProblemsFoundations of Computational Mathematics, 2012
- Latent Variable Bayesian Models for Promoting SparsityIEEE Transactions on Information Theory, 2011
- Contour Detection and Hierarchical Image SegmentationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2010
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998