Towards Understanding Hierarchical Learning: Benefits of Neural Representations
Preprint
Abstract
Deep neural networks can empirically perform efficient hierarchical learning, in which the layers learn useful representations of the data. However, how they make use of the intermediate representations are not explained by recent theories that relate them to "shallow learners" such as kernels. In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks and can be advantageous over raw inputs. We consider a fixed, randomly initialized neural network as a representation function fed into another trainable network. When the trainable network is the quadratic Taylor model of a wide two-layer network, we show that neural representation can achieve improved sample complexities compared with the raw input: For learning a low-rank degree-$p$ polynomial ($p \geq 4$) in $d$ dimension, neural representation requires only $\tilde{O}(d^{\lceil p/2 \rceil})$ samples, while the best-known sample complexity upper bound for the raw input is $\tilde{O}(d^{p-1})$. We contrast our result with a lower bound showing that neural representations do not improve over the raw input (in the infinite width limit), when the trainable network is instead a neural tangent kernel. Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Gradient descent optimizes over-parameterized deep ReLU networksMachine Learning, 2019
- Spectral Algorithms for Tensor CompletionCommunications on Pure and Applied Mathematics, 2018
- Error bounds for approximations with deep ReLU networksNeural Networks, 2017
- Introduction to the non-asymptotic analysis of random matrices2012
- Universal approximation bounds for superpositions of a sigmoidal functionIEEE Transactions on Information Theory, 1993
- Multilayer feedforward networks with a nonpolynomial activation function can approximate any functionNeural Networks, 1993
- Approximation by ridge functions and neural networks with one hidden layerJournal of Approximation Theory, 1992
- Approximation capabilities of multilayer feedforward networksNeural Networks, 1991
- Approximation by superpositions of a sigmoidal functionMathematics of Control, Signals, and Systems, 1989
- On the approximate realization of continuous mappings by neural networksNeural Networks, 1989