Characterising the Double Descent of Symbolic Regression
Open Access
- 14 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2050-2057
Abstract
Recent work has argued that many machine learning techniques exhibit a 'double descent' in model risk, where increasing model complexity beyond an interpolation zone can overcome the bias-variance tradeoff to produce large, over-parameterised models that generalise well to unseen data. While the double descent characteristic has been identified in many learning methods, it has not been explored within symbolic regression research. This paper presents an initial exploration into the presence of double descent behaviour in symbolic regression over a range of parameter settings. Results suggest that symbolic regression via genetic programming does not exhibit a clear double descent risk curve relative to model size or function set. Unlike other methods, models evolved through symbolic regression do not appear to strongly interpolate training data, which promotes a degree of robustness towards noise in training data. However, models evolved by symbolic regression can still be large and do not present a strong overfitting characteristic. Given that a prime motivation for symbolic regression is to produce compact interpretable models, these results suggest that methods aimed at regularising evolved models should be a key feature of all symbolic regression methods.Keywords
This publication has 31 references indexed in Scilit:
- Deep double descent: where bigger models and more data hurt*Journal of Statistical Mechanics: Theory and Experiment, 2021
- Pruning of genetic programming trees using permutation testsEvolutionary Intelligence, 2020
- Parameter identification for symbolic regression using nonlinear least squaresGenetic Programming and Evolvable Machines, 2019
- Reconciling modern machine-learning practice and the classical bias–variance trade-offProceedings of the National Academy of Sciences, 2019
- Structural Risk Minimization-Driven Genetic Programming for Enhancing Generalization in Symbolic RegressionIEEE Transactions on Evolutionary Computation, 2018
- Bloat and Generalisation in Symbolic Regression2014
- Numerical simplification for bloat control and analysis of building blocks in genetic programmingEvolutionary Intelligence, 2009
- Improving Symbolic Regression with Interval Arithmetic and Linear Scaling2003
- Genetic Programming and Domain Knowledge: Beyond the Limitations of Grammar-Guided Machine Discovery2000
- Genetic Programming, Ensemble Methods and the Bias/Variance Tradeoff – Introductory InvestigationsLecture Notes in Computer Science, 2000