MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training
Preprint
Abstract
The reliable application of deep learning models to software engineering tasks hinges on high-quality training data. Yet, large-scale repositories inevitably introduce noisy or mislabeled examples that degrade both accuracy and robustness. While Noise Label Learning (NLL) has been extensively studied in other fields, there are a few works that investigate NLL in Software Engineering (SE) and Large Language Models (LLMs) for SE tasks. In this work, we propose MANTRA, a Multi-stage Adaptive Noise TReAtment framework that embeds noise diagnosis and mitigation directly into the fine-tuning process of code-Pretrained Language Models (PTM) and code-LLMs. We first investigate the effect of noise at varying levels on convergence and loss trajectories of the models. Then we apply an adaptive dropout strategy guided by per-sample loss dynamics and Gaussian Mixture Model clustering to exclude persistently noisy points while preserving clean data. Applying to code summarization and commit intent classification, our experiments reveal that some LLMs are more sensitive to noise than others. However, with MANTRA, the performance of all models in both tasks is improved. MANTRA enables researchers and practitioners to reduce the impact of errors introduced by the dataset in training, saves time in data cleaning and processing, while maximizing the effect of fine-tuning.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- On the Effectiveness of LLM-as-a-Judge for Code Generation and SummarizationIEEE Transactions on Software Engineering, 2025
- Transforming Software Engineering Processes Through Generative AI: A Framework for Integration and ImplementationComputer, 2025
- SnipGen: A Mining Repository Framework for Evaluating LLMs for Code2025
- Facing & mitigating common challenges when working with real-world data: The Data Learning ParadigmJournal of Computational Science, 2025
- Robust Learning with Adversarial Perturbations and Label Noise2022
- Towards Using Data-Influence Methods to Detect Noisy Samples in Source Code Corpora2022
- Noisy label learning for security defects2022
- Multilingual training for software engineering2022
- Sketching Data Sets for Large-Scale Learning: Keeping only what you needIEEE Signal Processing Magazine, 2021
- Large-scale intent analysis for identifying large-review-effort code changesInformation and Software Technology, 2021