PSO-Convolutional Neural Networks With Heterogeneous Learning Rate
Open Access
- 23 August 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 89970-89988
Abstract
Convolutional Neural Networks (ConvNets or CNNs) have been candidly deployed in the scope of computer vision and related fields. Nevertheless, the dynamics of training of these neural networks lie still elusive: it is hard and computationally expensive to train them. A myriad of architectures and training strategies have been proposed to overcome this challenge and address several problems in image processing such as speech, image and action recognition as well as object detection. In this article, we propose a novel Particle Swarm Optimization (PSO) based training for ConvNets. In such framework, the vector of weights of each ConvNet is typically cast as the position of a particle in phase space whereby PSO collaborative dynamics intertwines with Stochastic Gradient Descent (SGD) in order to boost training performance and generalization. Our approach goes as follows: $i$ ) [regular phase] each ConvNet is trained independently via SGD; $ii$ ) [collaborative phase] ConvNets share among themselves their current vector of weights (or particle-position) along with their gradient estimates of the Loss function. Distinct step sizes are coined by distinct ConvNets. By properly blending ConvNets with large (possibly random) step-sizes along with more conservative ones, we propose an algorithm with competitive performance with respect to other PSO-based approaches on Cifar-10 and Cifar-100 (accuracy of 98.31% and 87.48%). These accuracy levels are obtained by resorting to only four ConvNets– such results are expected to scale with the number of collaborative ConvNets accordingly. We make our source codes available for download https://github.com/leonlha/ PSO-ConvNet-Dynamics.Keywords
All Related Versions
Funding Information
- FEDER Funds through the Programs COMPETE—Programa Operacional Factores de Competitividade and Centro2020—Centro Portugal Regional Operational Program
- national funds through FCT—Fundação para a Ciência e a Tecnologia (UIDB/00326/2020)
This publication has 36 references indexed in Scilit:
- An Analysis of the Inertia Weight Parameter for Binary Particle Swarm OptimizationIEEE Transactions on Evolutionary Computation, 2015
- Optimizing deep learning hyper-parameters through an evolutionary algorithm2015
- Going deeper with convolutions2015
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- Particle swarm optimization2014
- Practical Recommendations for Gradient-Based Training of Deep Architectures2012
- Inertia Weight strategies in Particle Swarm Optimization2011
- A novel particle swarm optimization algorithm with adaptive inertia weightApplied Soft Computing, 2011
- A modified particle swarm optimizer2002
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989