Miroslav Tůma

Our interest lies in the robust and efficient solution of large sparse linear least-squares problems. In recent years, hardware developments have led to a surge in interest in exploiting mixed-precision arithmetic within numerical linear algebra algorithms to take advantage of potential savings in memory requirements, runtime, and energy use, while still achieving the requested accuracy. We explore employing mixed precision when solving least-squares problems, focusing on the practicalities of developing robust approaches using low-precision incomplete Cholesky factorization preconditioners. Key penalties associated with lower precision include a loss of reliability and less accuracy in the computed solution. Through experiments involving problems from practical applications, we study computing incomplete Cholesky factorizations of the normal matrix using low precision and using the factors to precondition LSQR using mixed precision. We investigate level-based and memory-limited incomplete factorization preconditioners. We find that the former are not effective for least-squares problems while the latter can provide high-quality preconditioners. In particular, half precision arithmetic can be considered if high accuracy is not required in the solution or the memory for the incomplete factors is very restricted; otherwise, single precision can be used, and double-precision accuracy recovered while reducing memory consumption, even for ill-conditioned problems.

Least-squares problems are a cornerstone of computational science and engineering. Over the years, the size of the problems that researchers and practitioners face has constantly increased, making it essential that sparsity is exploited in the solution process. The goal of this article is to present a broad review of key algorithms for solving large-scale linear least-squares problems. This includes sparse direct methods and algebraic preconditioners that are used in combination with iterative solvers. Where software is available, this is highlighted.

Our interest lies in the robust and efficient solution of large sparse linear least-squares problems. In recent years, hardware developments have led to a surge in interest in exploiting mixed precision arithmetic within numerical linear algebra algorithms to take advantage of potential savings in memory requirements, runtime and energy use, whilst still achieving the requested accuracy. We explore employing mixed precision when solving least-squares problems, focusing on the practicalities of developing robust approaches using low precision incomplete Cholesky factorization preconditioners. Key penalties associated with lower precision include a loss of reliability and less accuracy in the computed solution. Through experiments involving problems from practical applications, we study computing incomplete Cholesky factorizations of the normal matrix using low precision and using the factors to precondition LSQR using mixed precision. We investigate level-based and memory-limited incomplete factorization preconditioners. We find that the former are not effective for least-squares problems while the latter can provide high-quality preconditioners. In particular, half precision arithmetic can be considered if high accuracy is not required in the solution or the memory for the incomplete factors is very restricted; otherwise, single precision can be used, and double precision accuracy recovered while reducing memory consumption, even for ill-conditioned problems.

Incomplete factorizations have long been popular general-purpose algebraic preconditioners for solving large sparse linear systems of equations. Guaranteeing the factorization is breakdown free while computing a high quality preconditioner is challenging. A resurgence of interest in using low precision arithmetic makes the search for robustness more important and more challenging. In this paper, we focus on ill-conditioned symmetric positive definite problems and explore a number of approaches for preventing and handling breakdowns: prescaling of the system matrix, a look-ahead strategy to anticipate breakdown as early as possible, the use of global shifts, and a modification of an idea developed in the field of numerical optimization for the complete Cholesky factorization of dense matrices. Our numerical simulations target highly ill-conditioned sparse linear systems with the goal of computing the factors in half precision arithmetic and then achieving double precision accuracy using mixed precision refinement. We also consider the often overlooked issue of growth in the sizes of entries in the factors that can occur when using any precision and can render the computed factors ineffective as preconditioners.

The emergence of low precision floating-point arithmetic in computer hardware has led to a resurgence of interest in the use of mixed precision numerical linear algebra. For linear systems of equations, there has been renewed enthusiasm for mixed precision variants of iterative refinement. We consider the iterative solution of large sparse systems using incomplete factorization preconditioners. The focus is on the robust computation of such preconditioners in half precision arithmetic and employing them to solve symmetric positive definite systems to higher precision accuracy; however, the proposed ideas can be applied more generally. Even for well-conditioned problems, incomplete factorizations can break down when small entries occur on the diagonal during the factorization. When using half precision arithmetic, overflows are an additional possible source of breakdown. We examine how breakdowns can be avoided and implement our strategies within new half precision Fortran sparse incomplete Cholesky factorization software. Results are reported for a range of problems from practical applications. These demonstrate that, even for highly ill-conditioned problems, half precision preconditioners can potentially replace double precision preconditioners, although unsurprisingly this may be at the cost of additional iterations of a Krylov solver.

The emergence of low precision floating-point arithmetic in computer hardware has led to a resurgence of interest in the use of mixed precision numerical linear algebra. For linear systems of equations, there has been renewed enthusiasm for mixed precision variants of iterative refinement. We consider the iterative solution of large sparse systems using incomplete factorization preconditioners. The focus is on the robust computation of such preconditioners in half precision arithmetic and employing them to solve symmetric positive definite systems to higher precision accuracy; however, the proposed ideas can be applied more generally. Even for well-conditioned problems, incomplete factorizations can break down when small entries occur on the diagonal during the factorization. When using half precision arithmetic, overflows are an additional possible source of breakdown. We examine how breakdowns can be avoided and we implement our strategies within new half precision Fortran sparse incomplete Cholesky factorization software. Results are reported for a range of problems from practical applications. These demonstrate that, even for highly ill-conditioned problems, half precision preconditioners can potentially replace double precision preconditioners, although unsurprisingly this may be at the cost of additional iterations of a Krylov solver.

In the field of recommender systems, shallow autoencoders have recently gained significant attention. One of the most highly acclaimed shallow autoencoders is easer, favored for its competitive recommendation accuracy and simultaneous simplicity. However, the poor scalability of easer (both in time and especially in memory) severely restricts its use in production environments with vast item sets. In this paper, we propose a hyperefficient factorization technique for sparse approximate inversion of the data-Gram matrix used in easer. The resulting autoencoder, sansa, is an end-to-end sparse solution with prescribable density and almost arbitrarily low memory requirements — even for training. As such, sansa allows us to effortlessly scale the concept of easer to millions of items and beyond.

We consider the problem of solving large-scale linear least squares problems that have one or more linear constraints that must be satisfied exactly. While some classical approaches are theoretically well founded, they can face difficulties when the matrix of constraints contains dense rows or if an algorithmic transformation used in the solution process results in a modified problem that is much denser than the original one. We propose modifications with an emphasis on requiring that the constraints be satisfied with a small residual. We examine combining the null-space method with our recently developed algorithm for computing a null-space basis matrix for a “wide” matrix. We further show that a direct elimination approach enhanced by careful pivoting can be effective in transforming the problem to an unconstrained sparse-dense least squares problem that can be solved with existing direct or iterative methods. We also present a number of solution variants that employ an augmented system formulation, which can be attractive for solving a sequence of related problems. Numerical experiments on problems coming from practical applications are used throughout to demonstrate the effectiveness of the different approaches.

Null-space methods have long been used to solve large sparse n × n symmetric saddle point systems of equations in which the (2, 2) block is zero. This paper focuses on the case where the (1, 1) block is ill conditioned or rank deficient and the k × k (2, 2) block is non zero and small (k ≪ n). Additionally, the (2, 1) block may be rank deficient. Such systems arise in a range of practical applications. A novel null-space approach is proposed that transforms the system matrix into a nicer symmetric saddle point matrix of order n that has a non zero (2, 2) block of order at most 2k and, importantly, the (1, 1) block is symmetric positive definite. Success of any null-space approach depends on constructing a suitable null-space basis. We propose methods for wide matrices having far fewer rows than columns with the aim of balancing stability of the transformed saddle point matrix with preserving sparsity in the (1, 1) block. Linear least squares problems that contain a small number of dense rows are an important motivation and are used to illustrate our ideas and to explore their potential for solving large-scale systems.