Mad Max: Affine Spline Insights Into Deep Learning
- 17 December 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Proceedings of the IEEE
- Vol. 109 (5) , 704-727
Abstract
We build a rigorous bridge between deep networks (DNs) and approximation theory via spline functions and operators. Our key result is that a large class of DNs can be written as a composition of max-affine spline operators (MASOs) that provide a powerful portal through which we view and analyze their inner workings. For instance, conditioned on the spline partition region containing the input signal, the output of an MASO DN can be written as a simple affine transformation of the input. This implies that a DN constructs a set of signal-dependent, class-specific templates against which the signal is compared via a simple inner product; we explore the links to the classical theory of optimal classification via matched filters and the effects of data memorization. Going further, we propose a simple penalty term that can be added to the cost function of any DN learning algorithm to force the templates to be orthogonal with each other; this leads to significantly improved classification performance and reduced overfitting with no change to the DN architecture. The spline partition of the input signal space that is implicitly induced by an MASO directly links DNs to the theory of vector quantization (VQ) and K-means clustering, which opens up new geometric avenues to study how DNs organize signals in a hierarchical fashion. To validate the utility of the VQ interpretation, we develop and validate a new distance metric for signals and images that quantify the difference between their VQ encodings.Keywords
All Related Versions
Funding Information
- K2I Graduate Fellowship (BP) from Rice University
- NSF (CCF-1911094, IIS-1838177, IIS-1730574)
- Army Research Office (W911NF-15-1-0316)
- Air Force Office of Scientific Research (FA9550-14-1-0088, FA9550-18-1-0478)
- Office of Naval Research (N00014-17-1-2551, N00014-18-12571)
- Defense Advanced Research Projects Agency (G001534-7500)
- DOD Vannevar Bush Faculty Fellowship of ONR (N00014-18-1-2047)
This publication has 34 references indexed in Scilit:
- Data clustering: 50 years beyond K-meansPattern Recognition Letters, 2010
- Generalization of Hinging HyperplanesIEEE Transactions on Information Theory, 2005
- Region configurations for realizability of lattice Piecewise-Linear modelsMathematical and Computer Modelling, 1999
- Flat MinimaNeural Computation, 1997
- Hinging hyperplanes for regression, classification, and function approximationIEEE Transactions on Information Theory, 1993
- Multilayer perceptron, fuzzy sets, and classificationIEEE Transactions on Neural Networks, 1992
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989
- Image coding using vector quantization: a reviewIEEE Transactions on Communications, 1988
- Learning representations by back-propagating errorsNature, 1986
- Structural shape optimization with geometric description and adaptive mesh refinementAIAA Journal, 1985