Conda: Column-Normalized Adam for Training Large Language Models Faster
Preprint
Abstract
Large language models (LLMs) have demonstrated impressive generalization and emergent capabilities, yet their pre-training remains computationally expensive and sensitive to optimization dynamics. While Adam-based optimizers offer fast convergence by adapting learning rates coordinate-wise, recent studies reveal that their updates often suffer from poor spectral conditioning and low-rank structures, hindering efficiency. Muon addresses this issue via global spectral normalization but lacks the per-coordinate adaptivity of Adam. In this work, we propose \textbf{Column-Normalized Adam (Conda)}, a novel optimizer that bridges the strengths of both approaches. Conda projects updates into an orthogonal subspace and applies column-wise second moment normalization based on the projected gradients, thereby achieving both improved spectral conditioning and maintaining coordinate-wise adaptivity. This design alleviates the spectral pathologies of Adam while preserving its fast convergence behavior. Extensive experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training. Remarkably, on the LLaMA series, \textbf{Conda achieves $2{\sim}2.5\times$ the convergence speed of AdamW, measured in both training steps and training time.} Further ablations demonstrate its robustness under diverse training setups. These results collectively highlight Conda as an effective and broadly applicable optimizer for large-scale LLM training. The code is released on https://github.com/jie040109/Conda
Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep ModelsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Low-Rank Gradient DescentIEEE Open Journal of Control Systems, 2023
- LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models2023
- MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms2019
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions2019
- Social IQa: Commonsense Reasoning about Social Interactions2019
- HellaSwag: Can a Machine Really Finish Your Sentence?2019
- Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering2018
- Optimization Methods for Large-Scale Machine LearningSIAM Review, 2018
- Crowdsourcing Multiple Choice Science Questions2017