MSMikhail Smelyanskiy
Mikhail Smelyanskiy
*This is an algorithmically generated author record based on publicly available information.
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Research Interests
Mikhail Smelyanskiy has a total of 9 co-authors. Here are the top co-authors by number of shared publications:
- Brian Nickerson
Intel Architecture Group
1 shared publication - Dheevatsa Mudigere
Intel Corporation Bangalore India
1 shared publication - Jorge Nocedal
Department of Industrial Engineering and Management Sciences Northwestern University Evan- ston IL USA
1 shared publication - Michael Chuvelev
Software and Solutions Group
1 shared publication - Michael Deisher
Intel Corporation Hillsboro Oregon USA
1 shared publication - Nitish Shirish Keskar
Department of Industrial Engineering and Management Sciences Northwestern University Evanston IL -60208 USA
1 shared publication
Latest Publications
- Open Access
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap.Open AccessOn Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data, say $32$-$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions - and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We discuss several strategies to attempt to help large-batch methods eliminate this generalization gap. - 14 April 2011
- Published by Springer Nature in Computer Science - Research and Development
- Vol. 26 (3-4) , 211-220
Designing and dynamically load balancing hybrid LU for multi/many-core
- 14 April 2011
- Published by Springer Nature in Computer Science - Research and Development
- Vol. 26 (3-4) , 211-220
Designing and dynamically load balancing hybrid LU for multi/many-core