Configuration Factors for Compact LLM‐Based Story Point Estimation: A Controlled Project‐Level Empirical Study
Open Access
- 1 January 2026
- journal article
- Published by Institution of Engineering and Technology (IET) in IET Software
- Vol. 2026 (1)
Abstract
Story point estimation is widely used in agile software planning, yet automated estimation remains challenging because issue descriptions and story point distributions vary substantially across projects. This study examines configuration sensitivity and comparative performance in compact large language model (LLM)‐based story point estimation through a controlled project‐level study using five TAWOS projects and two ~1–1.5B‐parameter compact backbones with frozen parameters and trainable prediction heads. The study comprises four‐factor configuration analysis covering maximum sequence length, metadata concatenation, learning rate, and soft‐target mixing; metadata‐free sensitivity analysis; learning‐rate–soft‐target‐mixing interaction analysis; comparison with three aligned baselines; assessment of alternative temperature‐selection objectives; and computational‐cost evaluation. Soft‐target mixing and learning rate show the larger average mean absolute error (MAE) effects among the four evaluated factors, with mean high‐minus‐low effects of −0.0535 and −0.0458, respectively, although both vary in direction across project–backbone–seed scenarios. The metadata‐free analysis shows similar average effect magnitudes for learning rate and soft‐target mixing, while the interaction between these two factors is heterogeneous, with 11 negative and 9 positive scenario‐level contrasts. When compact configurations and baseline methods are selected independently using seed‐averaged validation MAE, the corresponding test results favor the compact configuration in three projects and the selected baseline in two, with additional project‐specific variation among TFIDF‐LGBM, MiniLM‐MLP, and CodeBERT‐FT. Temperature‐selection rules produce different MAE, snapped‐expected calibration error (ECE), and negative log‐likelihood (NLL) profiles. The compact configurations require very small trainable prediction heads but retain substantially higher inference latency and memory requirements than the evaluated baselines; method‐level mean raw‐text‐to‐logits latency ranges ~0.37–25.72 ms/sample in the reported NVIDIA A10 environment. Overall, the results characterize compact‐LLM story point estimation by heterogeneous configuration sensitivity, project‐dependent model ranking, and accuracy–cost trade‐offs rather than a single consistently preferable configuration or model family.Keywords
This publication has 35 references indexed in Scilit:
- Explainable AI–driven optimization for transparent and auditable decision-making in medical softwareJournal of King Saud University Computer and Information Sciences, 2026
- Identification and empirical investigation of factors influencing cost estimation in Agile developmentApplied Intelligence, 2026
- Improving story points estimation using ensemble machine learningSoftware Quality Journal, 2025
- Machine learning experiment management tools: a mixed-methods empirical studyEmpirical Software Engineering, 2024
- Search-Based Optimisation of LLM Learning Shots for Story Point Estimation2023
- AIBugHunter: A Practical tool for predicting, classifying and repairing software vulnerabilitiesEmpirical Software Engineering, 2023
- Design of experiments and machine learning with application to industrial experimentsStatistical Papers, 2023
- Data-driven effort estimation techniques of agile user stories: a systematic literature reviewArtificial Intelligence Review, 2022
- Practical age estimation using deep label distribution learningFrontiers of Computer Science, 2020
- Challenges to the Reproducibility of Machine Learning Models in Health CareJAMA, 2020