Abstract
Nested cross‐validation is widely used for evaluating hyperparameter‐tuning strategies, but its effectiveness depends strongly on the choice of fold numbers, particularly the outer and inner. This issue is especially important in deep neural network tuning, where performance estimates may be sensitive to both data partitioning and hyperparameter search strategy. This paper investigates the effect of different outer‐ and inner‐fold values in nested k ‐fold cross‐validation for neural network models in both regression and classification tasks. Furthermore, a design of experiments approach, specifically the Plackett–Burman design, is employed as a screening baseline, and Bayesian optimization is used as an adaptive search strategy to generate candidate hyperparameter configurations for comparison under nested cross‐validation. By comparing these methodologies for hyperparameter tuning and model optimization, the study is aimed at providing practical guidance for improving the performance and reliability of hyperparameter tuning under nested cross‐validation. The results show that within the context of this study, predictive performance tends to improve as the inner‐fold value increases, although the gains become smaller at higher values and require substantially greater computational cost. Statistical analysis based on paired outer‐fold results demonstrates that Bayesian optimization significantly outperforms the Plackett–Burman design for white wine regression and red wine classification across all experimented inner‐fold values. For the obesity dataset, the relative advantage depends on the value of the inner fold. In particular, Bayesian optimization performs better at low inner‐fold values; the two methods are statistically not different at intermediate values; and the Plackett–Burman design performs better at larger inner‐fold values.

This publication has 35 references indexed in Scilit: