Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms
Preprint
Abstract
When training neural networks with custom objectives, such as ranking losses and shortest-path losses, a common problem is that they are, per se, non-differentiable. A popular approach is to continuously relax the objectives to provide gradients, enabling learning. However, such differentiable relaxations are often non-convex and can exhibit vanishing and exploding gradients, making them (already in isolation) hard to optimize. Here, the loss function poses the bottleneck when training a deep neural network. We present Newton Losses, a method for improving the performance of existing hard to optimize losses by exploiting their second-order information via their empirical Fisher and Hessian matrices. Instead of training the neural network with second-order techniques, we only utilize the loss function's second-order information to replace it by a Newton Loss, while training the network with gradient descent. This makes our method computationally efficient. We apply Newton Losses to eight differentiable algorithms for sorting and shortest-paths, achieving significant improvements for less-optimized differentiable algorithms, and consistent improvements, even for well-optimized differentiable algorithms.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Super-Universal Regularized Newton MethodSIAM Journal on Optimization, 2024
- Differentiable Ranking Metric Using Relaxed Sorting for Top-K RecommendationIEEE Access, 2021
- Weakly Supervised Person Re-ID: Differentiable Graphical Learning and a New BenchmarkIEEE Transactions on Neural Networks and Learning Systems, 2020
- Natural gradient via optimal transportInformation Geometry, 2018
- Newton Sketch: A Near Linear-Time Optimization Algorithm with Linear-Quadratic ConvergenceSIAM Journal on Optimization, 2017
- Perturbation Techniques in Online Learning and Optimization2016
- A Stochastic Quasi-Newton Method for Large-Scale OptimizationSIAM Journal on Optimization, 2016
- The Art of Computer Programming, Vol. 3: Sorting and SearchingMathematics of Computation, 1974