Baixi Sun
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Baixi Sun has a total of 29 co-authors. Here are the top co-authors by number of shared publications:
- Dingwen Tao
Chinese Academy of Sciences
3 shared publications - Jiannan Tian
Oakland University, Rochester, USA
3 shared publications - Xiaodong Yu
Stevens Institute of Technology, Hoboken, New Jersey, USA
3 shared publications - Chengming Zhang
University of Houston, Houston, TX, USA
2 shared publications - Daoce Wang
University of Nebraska Omaha, Omaha, Nebraska, USA
2 shared publications - Jinda Jia
Indiana University, Bloomington, USA
2 shared publications
Latest Publications
- Open Access
- 15 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 2038-2055
STZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data
- 15 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 2038-2055
Open AccessSTZ: A High Quality and High Speed Streaming Lossy Compression Framework for Scientific Data
- Open Access
- 28 February 2025
- Published by Association for Computing Machinery (ACM)
- p. 212-224
COMPSO: Optimizing Gradient Compression for Distributed Training with Second-Order Optimizers
Second-order optimization methods have been developed to enhance convergence and generalization in deep neural network (DNN) training compared to first-order methods like Stochastic Gradient Descent (SGD). However, these methods face challenges in distributed settings due to high communication overhead. Gradient compression, a technique commonly used to accelerate communication for first-order approaches, often results in low communication reduction ratios, decreased model accuracy, and/or high compression overhead when applied to second-order methods. To address these limitations, we introduce a novel gradient compression method for second-order optimizers called COMPSO. This method effectively reduces communication costs while preserving the advantages of second-order optimization. COMPSO employs stochastic rounding to maintain accuracy and filters out minor gradients to improve compression ratios. Additionally, we develop GPU optimizations to minimize compression overhead and performance modeling to ensure end-to-end performance gains across various systems. Evaluation of COMPSO on different DNN models shows that it achieves a compression ratio of 22.1×, reduces communication time by 14.2×, and improves overall performance by 1.9×, all without any drop in model accuracy.
- 28 February 2025
- Published by Association for Computing Machinery (ACM)
- p. 212-224
Open AccessCOMPSO: Optimizing Gradient Compression for Distributed Training with Second-Order Optimizers
Second-order optimization methods have been developed to enhance convergence and generalization in deep neural network (DNN) training compared to first-order methods like Stochastic Gradient Descent (SGD). However, these methods face challenges in distributed settings due to high communication overhead. Gradient compression, a technique commonly used to accelerate communication for first-order approaches, often results in low communication reduction ratios, decreased model accuracy, and/or high compression overhead when applied to second-order methods. To address these limitations, we introduce a novel gradient compression method for second-order optimizers called COMPSO. This method effectively reduces communication costs while preserving the advantages of second-order optimization. COMPSO employs stochastic rounding to maintain accuracy and filters out minor gradients to improve compression ratios. Additionally, we develop GPU optimizations to minimize compression overhead and performance modeling to ensure end-to-end performance gains across various systems. Evaluation of COMPSO on different DNN models shows that it achieves a compression ratio of 22.1×, reduces communication time by 14.2×, and improves overall performance by 1.9×, all without any drop in model accuracy.
- Open Access
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1759-1766
Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors
Transformer models have achieved remarkable success in various machine learning tasks but suffer from high computational complexity and resource requirements. The quadratic complexity of the self-attention mechanism further exacerbates these challenges when dealing with long sequences and large datasets. Specialized AI hardware accelerators, such as the Habana GAUDI architecture, offer a promising solution to tackle these issues. GAUDI features a Matrix Multiplication Engine (MME) and a cluster of fully programmable Tensor Processing Cores (TPC). This paper explores the untapped potential of using GAUDI processors to accelerate Transformer-based models, addressing key challenges in the process. Firstly, we provide a comprehensive performance comparison between the MME and TPC components, illuminating their relative strengths and weaknesses. Secondly, we explore strategies to optimize MME and TPC utilization, offering practical insights to enhance computational efficiency. Thirdly, we evaluate the performance of Transformers on GAUDI, particularly in handling long sequences and uncovering performance bottlenecks. Lastly, we evaluate the end-to-end performance of two Transformer-based large language models (LLM) on GAUDI. The contributions of this work encompass practical insights for practitioners and researchers alike. We delve into GAUDI’s capabilities for Transformers through systematic profiling, analysis, and optimization exploration. Our study bridges a research gap and offers a roadmap for optimizing Transformer-based model training on the GAUDI architecture.
- 12 November 2023
- Published by Association for Computing Machinery (ACM)
- p. 1759-1766
Open AccessBenchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processors
Transformer models have achieved remarkable success in various machine learning tasks but suffer from high computational complexity and resource requirements. The quadratic complexity of the self-attention mechanism further exacerbates these challenges when dealing with long sequences and large datasets. Specialized AI hardware accelerators, such as the Habana GAUDI architecture, offer a promising solution to tackle these issues. GAUDI features a Matrix Multiplication Engine (MME) and a cluster of fully programmable Tensor Processing Cores (TPC). This paper explores the untapped potential of using GAUDI processors to accelerate Transformer-based models, addressing key challenges in the process. Firstly, we provide a comprehensive performance comparison between the MME and TPC components, illuminating their relative strengths and weaknesses. Secondly, we explore strategies to optimize MME and TPC utilization, offering practical insights to enhance computational efficiency. Thirdly, we evaluate the performance of Transformers on GAUDI, particularly in handling long sequences and uncovering performance bottlenecks. Lastly, we evaluate the end-to-end performance of two Transformer-based large language models (LLM) on GAUDI. The contributions of this work encompass practical insights for practitioners and researchers alike. We delve into GAUDI’s capabilities for Transformers through systematic profiling, analysis, and optimization exploration. Our study bridges a research gap and offers a roadmap for optimizing Transformer-based model training on the GAUDI architecture.
- Open Access
- 21 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 324-335
HEAT: A Highly Efficient and Affordable Training System for Collaborative Filtering Based Recommendation on CPUs
- 21 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 324-335
Open AccessHEAT: A Highly Efficient and Affordable Training System for Collaborative Filtering Based Recommendation on CPUs