Neural Velocity for hyperparameter tuning
Preprint
Abstract
Hyperparameter tuning, such as learning rate decay and defining a stopping criterion, often relies on monitoring the validation loss. This paper presents NeVe, a dynamic training approach that adjusts the learning rate and defines the stop criterion based on the novel notion of "neural velocity". The neural velocity measures the rate of change of each neuron's transfer function and is an indicator of model convergence: sampling neural velocity can be performed even by forwarding noise in the network, reducing the need for a held-out dataset. Our findings show the potential of neural velocity as a key metric for optimizing neural network training efficientlyKeywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Regressing Word and Sentence Embeddings for Low-Resource Neural Machine TranslationIEEE Transactions on Artificial Intelligence, 2022
- Improving Calibration and Out-of-Distribution Detection in Deep Models for Medical Image SegmentationIEEE Transactions on Artificial Intelligence, 2022
- EO-MTRNN: evolutionary optimization of hyperparameters for a neuro-inspired computational model of spatiotemporal learningBiological Cybernetics, 2020
- Evolutionary Optimization of Hyperparameters in Deep Learning Models2019
- Wide Residual Networks2016
- On Early Stopping in Gradient Descent LearningConstructive Approximation, 2007
- Improving model selection by nonconvergent methodsNeural Networks, 1993
- Temporal Evolution of Generalization during Learning in Linear NetworksNeural Computation, 1991