Non-Random Weight Initialisation in Deep Convolutional Networks Applied to Safety Critical Artificial Intelligence
Authors:
- 14 December 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-8
Abstract
This paper presents a non-random weight initialisation scheme for convolutional neural network layers. It builds upon previous work that was limited to perceptron layers, but in that work repeatable determinism was achieved with equality in categorisation accuracy between the established random scheme and a linear ramp non-random scheme. This work however, is in Convolutional layers and are the layers that have been responsible for better than human performance in image recognition. The previous perceptron work found that number range was more important rather than the gradient. However, that was due to the fully connected nature of dense layers. Although, in convolutional layers by contrast, there is an order direction implied, and the weights relate to filters rather than image pixel positions, so the weight initialisation is more complex. However, the paper demonstrates a better performance, over the currently established random schemes with convolutional layers. The proposed method also induces earlier learning through the use of striped forms, and as such has less unlearning of the traditionally speckled random forms. That proposed scheme also provides a higher performing accuracy in a single learning session, with improvements of: 3.35% un-shuffled, 2.813% shuffled in the first epoch and 0.521% over the 5 epochs of the model. Of which the first epoch is more relevant as it is the epoch after initialisation. Also the proposed method is repeatable and deterministic, which is also a desirable quality for safety critical applications within image classification. The proposed method is also robust to He initialisation values too, and scored 97.55% accuracy compared to 96.929% accuracy with the Glorot/ Xavier in the traditional random forms, of which the benchmark model was originally optimised with.Keywords
This publication has 12 references indexed in Scilit:
- Repeatable determinism using non-random weight initialisations in smart city applications of deep learningJournal of Reliable Intelligent Environments, 2020
- Numerical Discrimination of the Generalisation Model from Learnt Weights in Neural Networks2019
- Non-Random Weight Initialisation in Deep Learning Networks for Repeatable Determinism2019
- Designing Safety Critical Software Systems to Manage Inherent Uncertainty2019
- ImageNet classification with deep convolutional neural networksCommunications of the ACM, 2017
- Safety and security in smart cities using artificial intelligence — A review2017
- Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction2011
- Convolutional networks and applications in vision2010
- Safety critical systems: challenges and directions2003
- Receptive fields and functional architecture of monkey striate cortexThe Journal of Physiology, 1968