Structure Learning for Deep Neural Networks Based on Multiobjective Optimization
- 5 May 2017
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 29 (6) , 2450-2463
Abstract
This paper focuses on the connecting structure of deep neural networks and proposes a layerwise structure learning method based on multiobjective optimization. A model with better generalization can be obtained by reducing the connecting parameters in deep networks. The aim is to find the optimal structure with high representation ability and better generalization for each layer. Then, the visible data are modeled with respect to structure based on the products of experts. In order to mitigate the difficulty of estimating the denominator in PoE, the denominator is simplified and taken as another objective, i.e., the connecting sparsity. Moreover, for the consideration of the contradictory nature between the representation ability and the network connecting sparsity, the multiobjective model is established. An improved multiobjective evolutionary algorithm is used to solve this model. Two tricks are designed to decrease the computational cost according to the properties of input data. The experiments on single-layer level, hierarchical level, and application level demonstrate the effectiveness of the proposed algorithm, and the learned structures can improve the performance of deep neural networks.Keywords
Funding Information
- National Natural Science Foundation of China (61422209)
- National Program for Support of Top-Notch Young Professionals of China
- Specialized Research Fund for the Doctoral Program of Higher Education (20130203110011)
This publication has 46 references indexed in Scilit:
- Representation Learning: A Review and New PerspectivesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
- Multiobjective Immune Algorithm with Nondominated Neighbor-Based SelectionEvolutionary Computation, 2008
- A Convolutional Neural Network Approach for Objective Video Quality AssessmentIEEE Transactions on Neural Networks, 2006
- A Fast Learning Algorithm for Deep Belief NetsNeural Computation, 2006
- Anatomical funneling, sparse connectivity and redundancy reduction in the neural networks of the basal gangliaJournal of Physiology-Paris, 2003
- Synchronization in Networks of Excitatory and Inhibitory Neurons with Sparse, Random ConnectivityNeural Computation, 2003
- Tuning of the structure and parameters of a neural network using an improved genetic algorithmIEEE Transactions on Neural Networks, 2003
- Training Products of Experts by Minimizing Contrastive DivergenceNeural Computation, 2002
- Evolving artificial neural networksProceedings of the IEEE, 1999
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998