YSYousef Saad

Yousef Saad

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Yousef Saad has a total of 96 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Graph Neural Multilevel Preconditioners for Iterative Solvers

    by Zechen Zhang, Rui Peng Li, Yousef Saad

  • Open Access

    Eigenvector-based acceleration strategies for gradient-type methods

    by Jean-Paul Chehab, Gaspard Kemlin, Marcos Raydan, Yousef Saad

    Several strategies are described and analyzed to speed-up gradient-type methods when applied to the minimization of strictly convex quadratics and strictly convex functions. The proposed techniques focus on relaxing the traditional optimal step length associated with gradient methods, including the steepest descent (SD) and the minimal residual (MR) methods. Such a relaxation avoids the well-known negative zigzag effect and allows the iterates to move in the entire space which in turn implies that every so often the search direction approaches some eigenvector of the underlying Hessian matrix. The proposed speedups then rely on taking advantage of the properties of the Lanczos method once a search direction that approaches an eigenvector has been identified in order to accelerate the convergence towards the global minimizer. After analyzing the proposed strategies, we illustrate them on the global minimization of strictly convex functions.

    • 16 January 2026
    • Published by HAL in HAL
    Open Access

    Eigenvector-based acceleration strategies for gradient-type methods

    by Jean-Paul Chehab, Gaspard Kemlin, Marcos Raydan, Yousef Saad

    Several strategies are described and analyzed to speed-up gradient-type methods when applied to the minimization of strictly convex quadratics and strictly convex functions. The proposed techniques focus on relaxing the traditional optimal step length associated with gradient methods, including the steepest descent (SD) and the minimal residual (MR) methods. Such a relaxation avoids the well-known negative zigzag effect and allows the iterates to move in the entire space which in turn implies that every so often the search direction approaches some eigenvector of the underlying Hessian matrix. The proposed speedups then rely on taking advantage of the properties of the Lanczos method once a search direction that approaches an eigenvector has been identified in order to accelerate the convergence towards the global minimizer. After analyzing the proposed strategies, we illustrate them on the global minimization of strictly convex functions.

  • Open Access

    Deep learning, transformers and graph neural networks: a linear algebra perspective

    by Abdelkader Baggag, Yousef Saad

    In an age where Artificial Intelligence (AI) is being integrated into nearly every domain of science and engineering, it has become essential for experts in Numerical Linear Algebra to explore the foundational elements of deep learning and identify ways to contribute to its development. What’s particularly exciting is that Numerical Linear Algebra (NLA) lies at the heart of Machine Learning and more broadly AI. All AI techniques fundamentally rely on four core components: data, optimization methods, statistical intuition, and linear algebra. The initial phase of any neural network model involves transforming the problem into one that can be tackled using numerical methods, particularly through optimization techniques. Thus, in Large Language Models (LLMs) this first step involves mapping words or subwords into tokens, which are then embedded into Euclidean spaces. From that point, LLMs rely heavily on vectors, matrices, and tensors. The aim of this article is to outline the essential components of deep learning methods from a linear algebra perspective. It will cover deep neural networks, multilayer perceptrons, and the concept of “attention,” which plays a crucial role in large language models as well as other machine learning applications. A significant portion of the discussion will focus on methods that leverage graphs in neural networks, such as Graph Convolutional Networks. The paper will conclude with reflections on the future role of numerical linear algebra in the age of AI.


  • Open Access

    Acceleration methods for fixed point iterations

    by Yousef Saad

    A pervasive approach in scientific computing is to express the solution to a given problem as the limit of a sequence of vectors or other mathematical objects. In many situations these sequences are generated by slowly converging iterative procedures and this led practitioners to seek faster alternatives to reach the limit. ``Acceleration techniques'' comprise a broad array of methods specifically designed with this goal in mind. They started as a means of improving the convergence of general scalar sequences by various forms of ``extrapolation to the limit'', i.e., by extrapolating the most recent iterates to the limit via linear combinations. Extrapolation methods of this type, the best known example of which is Aitken's Delta-squared process, require only the sequence of vectors as input. However, limiting methods to only use the iterates is too restrictive. Accelerating sequences generated by fixed-point iterations by utilizing both the iterates and the fixed-point mapping itself has proven highly successful across various areas of physics. A notable example of these Fixed-Point accelerators (FP-Accelerators) is a method developed by D. Anderson in 1965 and now widely known as Anderson Acceleration (AA). Furthermore, Quasi-Newton and Inexact Newton methods can also be placed in this category as well. This paper presents an overview of these methods -- with an emphasis on those, such as AA, that are geared toward accelerating fixed point iterations.

  • Open Access

    Acceleration methods for fixed-point iterations

    by Yousef Saad

    A pervasive approach in scientific computing is to express the solution to a given problem as the limit of a sequence of vectors or other mathematical objects. In many situations these sequences are generated by slowly converging iterative procedures, and this led practitioners to seek faster alternatives to reach the limit. ‘Acceleration techniques’ comprise a broad array of methods specifically designed with this goal in mind. They started as a means of improving the convergence of general scalar sequences by various forms of ‘extrapolation to the limit’, i.e. by extrapolating the most recent iterates to the limit via linear combinations. Extrapolation methods of this type, the best-known of which is Aitken’s delta-squared process, require only the sequence of vectors as input.However, limiting methods to use only the iterates is too restrictive. Accelerating sequences generated by fixed-point iterations by utilizing both the iterates and the fixed-point mapping itself has proved highly successful across various areas of physics. A notable example of these fixed-point accelerators (FP-accelerators) is a method developed by Donald Anderson in 1965 and now widely known as Anderson acceleration (AA). Furthermore, quasi-Newton and inexact Newton methods can also be placed in this category since they can be invoked to find limits of fixed-point iteration sequences by employing exactly the same ingredients as those of the FP-accelerators.This paper presents an overview of these methods – with an emphasis on those, such as AA, that are geared toward accelerating fixed-point iterations. We will navigate through existing variants of accelerators, their implementations and their applications, to unravel the close connections between them. These connections were often not recognized by the originators of certain methods, who sometimes stumbled on slight variations of already established ideas. Furthermore, even though new accelerators were invented in different corners of science, the underlying principles behind them are strikingly similar or identical.The plan of this article will approximately follow the historical trajectory of extrapolation and acceleration methods, beginning with a brief description of extrapolation ideas, followed by the special case of linear systems, the application to self-consistent field (SCF) iterations, and a detailed view of Anderson acceleration. The last part of the paper is concerned with more recent developments, including theoretical aspects, and a few thoughts on accelerating machine learning algorithms.


  • Open Access

    Mixed Precision Orthogonalization-Free Projection Methods for Eigenvalue and Singular Value Problems

    by Tianshi Xu, Zechen Zhang, Jie Chen, Yousef Saad, Yuanzhe Xi

    Mixed-precision arithmetic offers significant computational advantages for large-scale matrix computation tasks, yet preserving accuracy and stability in eigenvalue problems and the singular value decomposition (SVD) remains challenging. This paper introduces an approach that eliminates orthogonalization requirements in traditional Rayleigh-Ritz projection methods. The proposed method employs non-orthogonal bases computed at reduced precision, resulting in bases computed without inner-products. A primary focus is on maintaining the linear independence of the basis vectors. Through extensive evaluation with both synthetic test cases and real-world applications, we demonstrate that the proposed approach achieves the desired accuracy while fully taking advantage of mixed-precision arithmetic.

  • Open Access

    Joint Approximate Partial Diagonalization of Large Matrices

    by Abd-Krim Seghouane, Yousef Saad

    Given a set of $p$ symmetric (real) matrices, the Orthogonal Joint Diagonalization (OJD) problem consists of finding an orthonormal basis in which the representation of each of these $p$ matrices is as close as possible to a diagonal matrix. We argue that when the matrices are of large dimension, then the natural generalization of this problem is to seek an orthonormal basis of a certain subspace that is a near eigenspace for all the matrices in the set. We refer to this as the problem of ``partial joint diagonalization of matrices.'' The approach proposed first finds this approximate common near eigenspace and then proceeds to a joint diagonalization of the restrictions of the input matrices in this subspace. A few solution methods for this problem are proposed and illustrations of its potential applications are provided.

  • Open Access

    NLTGCR: A class of Nonlinear Acceleration Procedures based on Conjugate Residuals

    by Huan He, Ziyuan Tang, Shifan Zhao, Yousef Saad, Yuanzhe Xi

    This paper develops a new class of nonlinear acceleration algorithms based on extending conjugate residual-type procedures from linear to nonlinear equations. The main algorithm has strong similarities with Anderson acceleration as well as with inexact Newton methods - depending on which variant is implemented. We prove theoretically and verify experimentally, on a variety of problems from simulation experiments to deep learning applications, that our method is a powerful accelerated iterative algorithm.

  • Open Access

    On the tubular eigenvalues of third-order tensors

    by Fatemeh P. A. Beik, Yousef Saad

    This paper introduces the notion of tubular eigenvalues of third-order tensors with respect to T-products of tensors and analyzes their properties. A focus of the paper is to discuss relations between tubular eigenvalues and two alternative definitions of eigenvalue for third-order tensors that are known in the literature, namely eigentuple and T-eigenvalue. In addition, it establishes a few results on tubular spectra of tensors which can be exploited to analyze the convergence of tubular versions of iterative methods for solving tensor equations.