Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning
Top Cited Papers
- 1 September 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (9) , 10221-10234
Abstract
With the accumulation of resources in the era of big data and the rise of pre-trained models in deep learning, optimizing neural networks for various tasks often involves different strategies for fine-tuning pre-trained models versus training from scratch. However, existing optimizers primarily focus on reducing the loss function by updating model parameters, without fully addressing the unique demands of these two major paradigms. In this paper, we propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for these distinct training scenarios. For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization by aligning with the characteristics of weight updates and network architecture. For more importantly fine-tuning, we integrate weight rollback with the optimizer, incorporating a rollback term into each weight update step. This ensures consistency in the weight distribution between upstream and downstream models, effectively mitigating knowledge forgetting and improving fine-tuning performance. Additionally, we extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks. Extensive experiments across diverse tasks, including image classification, object detection, semantic segmentation, and instance segmentation, demonstrate the broad applicability and state-of-the-art performance of DualOpt. Code is available at https://github.com/qklee-lz/OLOR-AAAI-2024.Keywords
All Related Versions
Funding Information
- Beijing Natural Science Foundation (L233036)
- National Natural Science Foundation of China (62373343)
This publication has 45 references indexed in Scilit:
- RhythmFormer: Extracting patterned rPPG signals based on periodic sparse attentionPattern Recognition, 2025
- Weight Prediction Boosts the Convergence of AdamW2023
- Visual Prompt Tuning2022
- Practical Recommendations for Replay-Based Continual Learning Methods2022
- End-to-End Object Detection with Transformers2020
- The Open Images Dataset V4International Journal of Computer Vision, 2020
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015
- The SUN Attribute Database: Beyond Categories for Deeper Scene UnderstandingInternational Journal of Computer Vision, 2014
- Microsoft COCO: Common Objects in Context2014