Weilin Cai
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Weilin Cai has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Jiayi Huang
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
4 shared publications - Le Qin
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
3 shared publications Fan Wang
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
2 shared publications- Heng Chen
School of Computer Science and Technology, Xian Jiaotong University, Xian, China
2 shared publications - Junwei Cui
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
2 shared publications Juyong Jiang
The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China
2 shared publications
Latest Publications
- Open Access
- 17 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 659-674
Optimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks
- 17 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 659-674
Open AccessOptimizing All-to-All Collective Communication with Fault Tolerance on Torus Networks
- Open Access
- 20 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 498-513
Chimera: Communication Fusion for Hybrid Parallelism in Large Language Models
- 20 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 498-513
Open AccessChimera: Communication Fusion for Hybrid Parallelism in Large Language Models
- Open Access
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 655-671
MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
- 30 March 2025
- Published by Association for Computing Machinery (ACM)
- p. 655-671
Open AccessMoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
- Top Cited Papers
- 25 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. PP (10414347) , 1-20
A Survey on Mixture of Experts in Large Language Models
Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs.
- 25 March 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. PP (10414347) , 1-20
Top Cited PapersA Survey on Mixture of Experts in Large Language Models
Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts-in-LLMs.
- Open Access
- 9 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
A Survey on Mixture of Experts
Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository accessible at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts.
- 9 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in TechRxiv
Open AccessA Survey on Mixture of Experts
Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of LLMs is underpinned by their substantial model size, extensive and diverse datasets, and the vast computational power harnessed during training, all of which contribute to the emergent abilities of LLMs (e.g., in-context learning) that are not present in small models. Within this context, the mixture of experts (MoE) has emerged as an effective method for substantially scaling up model capacity with minimal computation overhead, gaining significant attention from academia and industry. Despite its growing prevalence, there lacks a systematic and comprehensive review of the literature on MoE. This survey seeks to bridge that gap, serving as an essential resource for researchers delving into the intricacies of MoE. We first briefly introduce the structure of the MoE layer, followed by proposing a new taxonomy of MoE. Next, we overview the core designs for various MoE models including both algorithmic and systemic aspects, alongside collections of available open-source implementations, hyperparameter configurations and empirical evaluations. Furthermore, we delineate the multifaceted applications of MoE in practice, and outline some potential directions for future research. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE research, we have established a resource repository accessible at https://github.com/withinmiaov/A-Survey-on-Mixture-of-Experts.
- 25 February 2022
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 78 (9) , 12046-12069
Extending $\tau $-Lop to model MPI blocking primitives on shared memory- 25 February 2022
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 78 (9) , 12046-12069
Extending $\tau $-Lop to model MPI blocking primitives on shared memory- 16 February 2022
- Published by Elsevier in Future Generation Computer Systems
- Vol. 132, 25-39
LogSC: Model-based one-sided communication performance estimation
- 16 February 2022
- Published by Elsevier in Future Generation Computer Systems
- Vol. 132, 25-39
LogSC: Model-based one-sided communication performance estimation
- 24 August 2021
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 78 (3) , 4199-4216
Implementation and optimization of ChaCha20 stream cipher on sunway taihuLight supercomputer
- 24 August 2021
- Published by Springer Nature in The Journal of Supercomputing
- Vol. 78 (3) , 4199-4216
Implementation and optimization of ChaCha20 stream cipher on sunway taihuLight supercomputer