Frank-Wolfe Network: An Interpretable Deep Structure for Non-Sparse Coding
- 19 August 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. 30 (9) , 3068-3080
Abstract
The problem of Lp-norm constrained coding is to convert signal into code that lies inside an Lp-ball and most faithfully reconstructs the signal. Previous works under the name of sparse coding considered the cases of L0 and L1 norms. The cases with p > 1 values, i.e. non-sparse coding studied in this paper, remain a difficulty. We propose an interpretable deep structure namely Frank-Wolfe Network (F-W Net), whose architecture is inspired by unrolling and truncating the Frank-Wolfe algorithm for solving an Lp-norm constrained problem with p ≥ 1. We show that the Frank-Wolfe solver for the Lp-norm constraint leads to a novel closed-form nonlinear unit, which is parameterized by p and termed poolp. The poolp unit links the conventional pooling, activation, and normalization operations, making F-W Net distinct from existing deep networks either heuristically designed or converted from projected gradient descent algorithms. We further show that the hyper-parameter p can be made learnable instead of pre-chosen in F-W Net, which gracefully solves the non-sparse coding problem even with unknown p. We evaluate the performance of F-W Net on an extensive range of simulations as well as the task of handwritten digit recognition, where F-W Net exhibits strong learning capability. We then propose a convolutional version of FWNet, and apply the convolutional F-W Net into image denoising and super-resolution tasks, where F-W Net all demonstrates impressive effectiveness, flexibility, and robustness.Keywords
All Related Versions
Funding Information
- National Key Research and Development Plan (2017YFB1002401)
- Natural Science Foundation of China (61772483)
This publication has 47 references indexed in Scilit:
- $L_{1/2}$ Regularization: A Thresholding Representation Theory and a Fast SolverIEEE Transactions on Neural Networks and Learning Systems, 2012
- Restricted isometry properties and nonconvex compressive sensingInverse Problems, 2008
- $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse RepresentationIEEE Transactions on Signal Processing, 2006
- For most large underdetermined systems of linear equations the minimal 𝓁1‐norm solution is also the sparsest solutionCommunications on Pure and Applied Mathematics, 2006
- Atomic Decomposition by Basis PursuitSIAM Review, 2001
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998
- Sparse coding with an overcomplete basis set: A strategy employed by V1?Vision Research, 1997
- Long Short-Term MemoryNeural Computation, 1997
- Sparse Approximate Solutions to Linear SystemsSIAM Journal on Computing, 1995
- An algorithm for quadratic programmingNaval Research Logistics Quarterly, 1956