Weight smoothing to improve network generalization
Authors:
- Jin WangJWJin Wang
- 1 January 1994
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 5 (5) , 752-763
Abstract
A weight smoothing algorithm is proposed in this paper to improve a neural network's generalization capability. The algorithm can be used when the data patterns to be classified are presented on an n-dimensional grid (n>/=1) and there exists some correlations among neighboring data points within a pattern. For a fully-interconnected feedforward net, no such correlation information is embedded into the architecture. Consequently, the correlations can only be extracted through sufficient amount of network training. With the proposed algorithm, a smoothing constraint is incorporated into the objective function of backpropagation to reflect the neighborhood correlations and to seek those solutions that have smooth connection weights. Experiments were performed on problems of waveform classification, multifont alphanumeric character recognition, and handwritten numeral recognition. The results indicate that (1) networks trained with the algorithm do have smooth connection weights, and (2) they generalize better.Keywords
This publication has 14 references indexed in Scilit:
- Translation, rotation, and scale invariant pattern recognition by high-order neural networks and moment classifiersIEEE Transactions on Neural Networks, 1992
- A study of methods of choosing the smoothing parameter in image restoration by regularizationIEEE Transactions on Pattern Analysis and Machine Intelligence, 1991
- Encoding a priori information in feedforward networksNeural Networks, 1991
- Backpropagation Applied to Handwritten Zip Code RecognitionNeural Computation, 1989
- Handwritten digit recognition: applications of neural network chips and automatic learningIEEE Communications Magazine, 1989
- A neural network approach to character recognitionNeural Networks, 1989
- A neural network for visual pattern recognitionComputer, 1988
- Neocognitron: A hierarchical neural network capable of visual pattern recognitionNeural Networks, 1988
- Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of ImagesIEEE Transactions on Pattern Analysis and Machine Intelligence, 1984
- Optimization by Simulated AnnealingScience, 1983