Junda Feng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Junda Feng has a total of 41 co-authors. Here are the top co-authors by number of shared publications:
- Haibin Lin
ByteDance Inc., Seattle, USA
3 shared publications - Jianyu Jiang
ByteDance Seed, Shanghai, China
2 shared publications - Lei Zuo
ByteDance Inc., Seattle, USA
2 shared publications - Xin Liu
ByteDance Inc., Seattle, USA
2 shared publications - Yanghua Peng
ByteDance Inc., Seattle, USA
2 shared publications - Yangrui Chen
ByteDance Inc., Shanghai, China
2 shared publications
Latest Publications
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-18
MegaScale-Data: Scaling DataLoader for Multisource Large Foundation Model Training
Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When preparing data for LFM training that originates from multiple, distinct sources, two fundamental challenges arise. First, due to the quadratic computational complexity of the attention operator, the non-uniform sample distribution over data-parallel ranks leads to significant workload imbalance among dataloaders, degrading the training efficiency. Second, supporting diverse data sources requires per-dataset file access states that are redundantly replicated across parallel loaders, consuming excessive memory. This also hinders dynamic data mixing (e.g., curriculum learning) and causes redundant access/memory overhead in hybrid parallelism. We present MegaScale-Data, an industrial-grade distributed data loading architecture for multisource LFMs training, with three key innovations: (1) Disaggregated data preprocessing via role-specific actors (Source Loaders/Data Constructors) to eliminate source and parallelism redundant data access and ensure multisource scalability. (2) Centralized and declarative data plane for load-time multisource orchestration, such as long-short context, multimodality, and curriculum learning. (3) Multi-level auto-partitioning and scaling mechanism for source loaders under heterogeneous preprocessing costs. We also contribute our designs and operational experience in deployment and fault tolerance. MegaScale-Data achieves up to: (1) 4.5X end-to-end training throughput improvement, and (2) 13.5X reduction in CPU memory usage.
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Open AccessMegaScale-Data: Scaling DataLoader for Multisource Large Foundation Model Training
Modern frameworks for training large foundation models (LFMs) employ dataloaders in a data-parallel manner, with each loader processing a disjoint subset of training data. When preparing data for LFM training that originates from multiple, distinct sources, two fundamental challenges arise. First, due to the quadratic computational complexity of the attention operator, the non-uniform sample distribution over data-parallel ranks leads to significant workload imbalance among dataloaders, degrading the training efficiency. Second, supporting diverse data sources requires per-dataset file access states that are redundantly replicated across parallel loaders, consuming excessive memory. This also hinders dynamic data mixing (e.g., curriculum learning) and causes redundant access/memory overhead in hybrid parallelism. We present MegaScale-Data, an industrial-grade distributed data loading architecture for multisource LFMs training, with three key innovations: (1) Disaggregated data preprocessing via role-specific actors (Source Loaders/Data Constructors) to eliminate source and parallelism redundant data access and ensure multisource scalability. (2) Centralized and declarative data plane for load-time multisource orchestration, such as long-short context, multimodality, and curriculum learning. (3) Multi-level auto-partitioning and scaling mechanism for source loaders under heterogeneous preprocessing costs. We also contribute our designs and operational experience in deployment and fault tolerance. MegaScale-Data achieves up to: (1) 4.5X end-to-end training throughput improvement, and (2) 13.5X reduction in CPU memory usage.
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 675-692
MegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate 1.27×–7.57× throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems.
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 675-692
Open AccessMegaScale-Omni: A Hyper-Scale, Workload-Resilient System for MultiModal LLM Training in Production
As the foundational component of versatile AI applications, training an multimodal large language model (MLLM) relies on multimodal datasets with dynamic modality mixture proportions and sample length distributions. However, existing MLLM systems remain inefficient under dynamic workloads, due to statically coupled decisions of resource allocation and model parallelization between encoders and the LLM backbone. This paper presents MegaScale-Omni, an industrial-grade MLLM training system tailored for dynamic workload adaption and hyper-scale deployment. MegaScale-Omni is built upon the training scheme of encoder-LLM multiplexing with three key innovations: (1) Decoupled parallelism strategies with long-short sequence parallelism for encoders to process variable-length samples, and full-fledged 5D parallelism for the LLM backbone, both organized under a communication-efficient parallelization layout. (2) Unified encoder-LLM representations for flexible, extensible colocation, and a new paradigm of encoder-LLM joint pipeline with workload resilience. (3) Workload balancing techniques via decentralized grouped reordering in data loaders and adaptive resharding from encoder to LLM ranks. MegaScale-Omni is deployed as the foundation of our in-house large-scale MLLM training tasks with thousands of GPUs. Our experimental results demonstrate 1.27×–7.57× throughput improvement under production-grade dynamic workloads, as compared to four state-of-the-art systems.
- Open Access
- 27 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 963-978
ByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs
- 27 August 2025
- Published by Association for Computing Machinery (ACM)
- p. 963-978
Open AccessByteScale: Communication-Efficient Scaling of LLM Training with a 2048K Context Length on 16384 GPUs
- 16 January 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Electromagnetic Compatibility
- Vol. 65 (2) , 436-453
Predicting Statistical Wave Physics in Complex Enclosures: A Stochastic Dyadic Green's Function Approach
This article presents a physics-oriented, mathematically tractable, statistical wave model for analyzing the wave physics of high-frequency reverberation in complex cavity environments. The key ingredient is a vector dyadic stochastic Green's function (SGF) method that is derived from the Wigner's random matrix theory and Berry's random wave hypothesis. The SGF statistically replicates multipath, ray-chaotic communication between vector sources and vectorial electromagnetic fields at displaced observation points using generic, macroscopic parameters of the cavity environment. The work establishes a physics-based modeling and simulation capability that predicts the probabilistic behavior of backdoor coupling to complex electronic enclosures. Experimental results are supplied to validate the proposed work.
- 16 January 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Electromagnetic Compatibility
- Vol. 65 (2) , 436-453
Predicting Statistical Wave Physics in Complex Enclosures: A Stochastic Dyadic Green's Function Approach
This article presents a physics-oriented, mathematically tractable, statistical wave model for analyzing the wave physics of high-frequency reverberation in complex cavity environments. The key ingredient is a vector dyadic stochastic Green's function (SGF) method that is derived from the Wigner's random matrix theory and Berry's random wave hypothesis. The SGF statistically replicates multipath, ray-chaotic communication between vector sources and vectorial electromagnetic fields at displaced observation points using generic, macroscopic parameters of the cavity environment. The work establishes a physics-based modeling and simulation capability that predicts the probabilistic behavior of backdoor coupling to complex electronic enclosures. Experimental results are supplied to validate the proposed work.