QualiTagger: automating software quality categorization in issue trackers
Open Access
- 17 July 2026
- journal article
- research article
- Published by Springer Nature in Software Quality Journal
- Vol. 34 (3) , 38
Abstract
Managing software quality is crucial for maintainable software systems, yet understanding how quality attributes are discussed within the informal, noisy context of issue trackers remains a significant challenge. Current automatic approaches for categorizing quality concerns often falter, as they are typically validated on small domain-specific datasets and are ill-equipped to parse the conversational language of developer discourse. This hinders effective prioritization and technical debt management. This paper introduces QualiTagger, an automated approach for classifying seven distinct software quality attributes from issue tracker text, and QualiDataSet, a novel, curated dataset of over 700,000 labeled GitHub issues that underpins this work. We demonstrate that an ensemble of specialized binary classifiers, built upon the DistilRoBERTa architecture, significantly outperforms a single multiclass model and shows superior or comparable efficacy to a general-purpose Large Language Model (GPT-4o) for this task. The model’s real-world applicability is further validated through an industrial case study at Visma focusing on security issues) and a user study with software engineering students. Our evaluation confirms that QualiTagger achieves high classification accuracy and, crucially, generalizes effectively to previously unseen (Out-of-Distribution) projects, a key indicator of its practical utility. By providing both a robust classification tool and a large-scale public dataset, this research enables a more nuanced, data-driven understanding of how software quality is managed in practice, offering valuable insights for project management and future empirical software engineering researchKeywords
All Related Versions
Funding Information
- University of Oslo
This publication has 26 references indexed in Scilit:
- BEACon-TD: Classifying Technical Debt and its types across diverse software projects issues using transformersJournal of Systems and Software, 2025
- The application of AI techniques in requirements classification: a systematic mappingArtificial Intelligence Review, 2024
- Automatically classifying non-functional requirements using deep neural networkPattern Recognition, 2022
- The Use of NLP-Based Text Representation Techniques to Support Requirement Engineering Tasks: A Systematic Mapping ReviewIEEE Access, 2022
- Software quality prediction using machine learningMaterials Today: Proceedings, 2022
- Comparative Analysis for Predicting Non-Functional Requirements using Supervised Machine Learning2021
- A comparison of quality flaws and technical debt in model transformation specificationsJournal of Systems and Software, 2020
- Automatic Multi-class Non-Functional Software Requirements Classification Using Neural Networks2019
- Backlog Grooming2018
- Technical Debt tracking: Current state of practiceScience of Computer Programming, 2018