Conda: Column-Normalized Adam for Training Large Language Models Faster
Preprint
Abstract
Large language models (LLMs) have demonstrated impressive generalization and emergent capabilities, yet their pre-training remains computationally expensive and sensitive to optimization dynamics. While Adam-based optimizers offer fast convergence by adapting learning rates coordinate-wise, recent studies reveal that their updates often suffer from poor spectral conditioning and low-rank structures, hindering efficiency. Muon addresses this issue via global spectral normalization but lacks the per-coordinate adaptivity of Adam. In this work, we propose Column-Normalized Adam (Conda), a novel optimizer that bridges the strengths of both approaches. Conda projects updates into an orthogonal subspace and applies column-wise second moment normalization based on the projected gradients, thereby achieving both improved spectral conditioning and maintaining coordinate-wise adaptivity. This design alleviates the spectral pathologies of Adam while preserving its fast convergence behavior. Extensive experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training. Remarkably, on the LLaMA series, Conda achieves 2-2.5 the convergence speed of AdamW, measured in both training steps and training time. Further ablations demonstrate its robustness under diverse training setups. These results collectively highlight Conda as an effective and broadly applicable optimizer for large-scale LLM training. The code is released on https://github.com/jie040109/CondaKeywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Low-Rank Gradient DescentIEEE Open Journal of Control Systems, 2023
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models2023
- MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms2019
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions2019
- Social IQa: Commonsense Reasoning about Social Interactions2019
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018
- Optimization Methods for Large-Scale Machine LearningSIAM Review, 2018
- Crowdsourcing Multiple Choice Science Questions2017