Stefan Coors
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Stefan Coors has a total of 47 co-authors. Here are the top co-authors by number of shared publications:
- Bernd Bischl
LMU Munich
4 shared publications - Michel Lang
TU Dortmund University
4 shared publications - Jakob Richter
Department of Statistics, TU Dortmund University, Dortmund, Germany
3 shared publications - Florian Pfisterer
Ludwig-Maximilians-Universität München
2 shared publications - Marc Becker
Department of Statistics, Ludwig Maximilian University of Munich, Munich, Germany
2 shared publications Martin Binder
Department of Statistics Ludwig‐Maximilians‐Universität München Munich Germany
2 shared publications
Latest Publications
- Open Access
- 12 December 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Evolutionary Learning and Optimization
- Vol. 3 (4) , 1-50
Multi-Objective Hyperparameter Optimization in Machine Learning—An Overview
Hyperparameter optimization constitutes a large part of typical modern machine learning (ML) workflows. This arises from the fact that ML methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies from the domains of evolutionary algorithms and Bayesian optimization. We illustrate the utility of multi-objective optimization in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability, and robustness.
- 12 December 2023
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Evolutionary Learning and Optimization
- Vol. 3 (4) , 1-50
Open AccessMulti-Objective Hyperparameter Optimization in Machine Learning—An Overview
Hyperparameter optimization constitutes a large part of typical modern machine learning (ML) workflows. This arises from the fact that ML methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies from the domains of evolutionary algorithms and Bayesian optimization. We illustrate the utility of multi-objective optimization in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability, and robustness.
- Open Access
- 28 July 2023
- Published by Wolters Kluwer Health in Investigative Radiology
- Vol. 58 (12) , 874-881
A Comprehensive Machine Learning Benchmark Study for Radiomics-Based Survival Analysis of CT Imaging Data in Patients With Hepatic Metastases of CRC
Objectives: Optimizing a machine learning (ML) pipeline for radiomics analysis involves numerous choices in data set composition, preprocessing, and model selection. Objective identification of the optimal setup is complicated by correlated features, interdependency structures, and a multitude of available ML algorithms. Therefore, we present a radiomics-based benchmarking framework to optimize a comprehensive ML pipeline for the prediction of overall survival. This study is conducted on an image set of patients with hepatic metastases of colorectal cancer, for which radiomics features of the whole liver and of metastases from computed tomography images were calculated. A mixed model approach was used to find the optimal pipeline configuration and to identify the added prognostic value of radiomics features. Materials and Methods: In this study, a large-scale ML benchmark pipeline consisting of preprocessing, feature selection, dimensionality reduction, hyperparameter optimization, and training of different models was developed for radiomics-based survival analysis. Portal-venous computed tomography imaging data from a previous prospective randomized trial evaluating radioembolization of liver metastases of colorectal cancer were quantitatively accessible through a radiomics approach. One thousand two hundred eighteen radiomics features of hepatic metastases and the whole liver were calculated, and 19 clinical parameters (age, sex, laboratory values, and treatment) were available for each patient. Three ML algorithms—a regression model with elastic net regularization (glmnet), a random survival forest (RSF), and a gradient tree-boosting technique (xgboost)—were evaluated for 5 combinations of clinical data, tumor radiomics, and whole-liver features. Hyperparameter optimization and model evaluation were optimized toward the performance metric integrated Brier score via nested cross-validation. To address dependency structures in the benchmark setup, a mixed-model approach was developed to compare ML and data configurations and to identify the best-performing model. Results: Within our radiomics-based benchmark experiment, 60 ML pipeline variations were evaluated on clinical data and radiomics features from 491 patients. Descriptive analysis of the benchmark results showed a preference for RSF-based pipelines, especially for the combination of clinical data with radiomics features. This observation was supported by the quantitative analysis via a linear mixed model approach, computed to differentiate the effect of data sets and pipeline configurations on the resulting performance. This revealed the RSF pipelines to consistently perform similar or better than glmnet and xgboost. Further, for the RSF, there was no significantly better-performing pipeline composition regarding the sort of preprocessing or hyperparameter optimization. Conclusions: Our study introduces a benchmark framework for radiomics-based survival analysis, aimed at identifying the optimal settings with respect to different radiomics data sources and various ML pipeline variations, including preprocessing techniques and learning algorithms. A suitable analysis tool for the benchmark results is provided via a mixed model approach, which showed for our study on patients with intrahepatic liver metastases, that radiomics features captured the patients' clinical situation in a manner comparable to the provided information solely from clinical parameters. However, we did not observe a relevant additional prognostic value obtained by these radiomics features.
- 28 July 2023
- Published by Wolters Kluwer Health in Investigative Radiology
- Vol. 58 (12) , 874-881
Open AccessA Comprehensive Machine Learning Benchmark Study for Radiomics-Based Survival Analysis of CT Imaging Data in Patients With Hepatic Metastases of CRC
Objectives: Optimizing a machine learning (ML) pipeline for radiomics analysis involves numerous choices in data set composition, preprocessing, and model selection. Objective identification of the optimal setup is complicated by correlated features, interdependency structures, and a multitude of available ML algorithms. Therefore, we present a radiomics-based benchmarking framework to optimize a comprehensive ML pipeline for the prediction of overall survival. This study is conducted on an image set of patients with hepatic metastases of colorectal cancer, for which radiomics features of the whole liver and of metastases from computed tomography images were calculated. A mixed model approach was used to find the optimal pipeline configuration and to identify the added prognostic value of radiomics features. Materials and Methods: In this study, a large-scale ML benchmark pipeline consisting of preprocessing, feature selection, dimensionality reduction, hyperparameter optimization, and training of different models was developed for radiomics-based survival analysis. Portal-venous computed tomography imaging data from a previous prospective randomized trial evaluating radioembolization of liver metastases of colorectal cancer were quantitatively accessible through a radiomics approach. One thousand two hundred eighteen radiomics features of hepatic metastases and the whole liver were calculated, and 19 clinical parameters (age, sex, laboratory values, and treatment) were available for each patient. Three ML algorithms—a regression model with elastic net regularization (glmnet), a random survival forest (RSF), and a gradient tree-boosting technique (xgboost)—were evaluated for 5 combinations of clinical data, tumor radiomics, and whole-liver features. Hyperparameter optimization and model evaluation were optimized toward the performance metric integrated Brier score via nested cross-validation. To address dependency structures in the benchmark setup, a mixed-model approach was developed to compare ML and data configurations and to identify the best-performing model. Results: Within our radiomics-based benchmark experiment, 60 ML pipeline variations were evaluated on clinical data and radiomics features from 491 patients. Descriptive analysis of the benchmark results showed a preference for RSF-based pipelines, especially for the combination of clinical data with radiomics features. This observation was supported by the quantitative analysis via a linear mixed model approach, computed to differentiate the effect of data sets and pipeline configurations on the resulting performance. This revealed the RSF pipelines to consistently perform similar or better than glmnet and xgboost. Further, for the RSF, there was no significantly better-performing pipeline composition regarding the sort of preprocessing or hyperparameter optimization. Conclusions: Our study introduces a benchmark framework for radiomics-based survival analysis, aimed at identifying the optimal settings with respect to different radiomics data sources and various ML pipeline variations, including preprocessing techniques and learning algorithms. A suitable analysis tool for the benchmark results is provided via a mixed model approach, which showed for our study on patients with intrahepatic liver metastases, that radiomics features captured the patients' clinical situation in a manner comparable to the provided information solely from clinical parameters. However, we did not observe a relevant additional prognostic value obtained by these radiomics features.
- Top Cited PapersOpen Access
- 16 January 2023
- Published by Wiley in WIREs Data Mining and Knowledge Discovery
- Vol. 13 (2)
Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time‐consuming and irreproducible manual process of trial‐and‐error to find well‐performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization.This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction
- 16 January 2023
- Published by Wiley in WIREs Data Mining and Knowledge Discovery
- Vol. 13 (2)
Top Cited PapersOpen AccessHyperparameter optimization: Foundations, algorithms, best practices, and open challenges
Most machine learning algorithms are configured by a set of hyperparameters whose values must be carefully chosen and which often considerably impact performance. To avoid a time‐consuming and irreproducible manual process of trial‐and‐error to find well‐performing hyperparameter configurations, various automatic hyperparameter optimization (HPO) methods—for example, based on resampling error estimation for supervised machine learning—can be employed. After introducing HPO from a general perspective, this paper reviews important HPO methods, from simple techniques such as grid or random search to more advanced methods like evolution strategies, Bayesian optimization, Hyperband, and racing. This work gives practical recommendations regarding important choices to be made when conducting HPO, including the HPO algorithms themselves, performance evaluation, how to combine HPO with machine learning pipelines, runtime improvements, and parallelization.This article is categorized under: Algorithmic Development > Statistics Technologies > Machine Learning Technologies > Prediction
- Open Access
- 30 July 2022
- Published by Center for Open Science in PsyArXiv
Machine learning for spelling acquisition - How accurate is the prediction of specific spelling errors in German primary school students?
In Germany (similar to other countries), 30 % of students demonstrate insufficient spelling skills at the end of primary school – partly owing to the challenge for teachers to manage a variety of students’ learning needs. Digital tools using Machine Learning can enable teachers to individualise students’ learning. However, there are still no suitable approaches for demographics of students who are not yet proficient in spelling.With an aim to adapt Machine Learning for students of all proficiencies, we investigate how accurately specific spelling errors can be predicted across different skill levels, and what the content-related reasons for incorrect predictions are.To that end, we developed a web application to record the spelling efforts of N = 685 first- and second-graders in Bavaria, Germany. A total of 18,133 different misspellings were recorded. Using this dataset, we trained six Machine Learning models and compared their performances to predict misspellings.Comparing all Machine Learning models employed in this work, the Random Forest performed best on average as a predictor of spelling errors. Errors at the syllable- and morpheme-levels were predicted best, and errors at the basic phoneme-grapheme-level were predicted slightly less accurately. Confusions often concerned cases that are considered linguistically ambiguous or occurred in complex error entanglements. The implications of these results are discussed.
- 30 July 2022
- Published by Center for Open Science in PsyArXiv
Open AccessMachine learning for spelling acquisition - How accurate is the prediction of specific spelling errors in German primary school students?
In Germany (similar to other countries), 30 % of students demonstrate insufficient spelling skills at the end of primary school – partly owing to the challenge for teachers to manage a variety of students’ learning needs. Digital tools using Machine Learning can enable teachers to individualise students’ learning. However, there are still no suitable approaches for demographics of students who are not yet proficient in spelling.With an aim to adapt Machine Learning for students of all proficiencies, we investigate how accurately specific spelling errors can be predicted across different skill levels, and what the content-related reasons for incorrect predictions are.To that end, we developed a web application to record the spelling efforts of N = 685 first- and second-graders in Bavaria, Germany. A total of 18,133 different misspellings were recorded. Using this dataset, we trained six Machine Learning models and compared their performances to predict misspellings.Comparing all Machine Learning models employed in this work, the Random Forest performed best on average as a predictor of spelling errors. Errors at the syllable- and morpheme-levels were predicted best, and errors at the basic phoneme-grapheme-level were predicted slightly less accurately. Confusions often concerned cases that are considered linguistically ambiguous or occurred in complex error entanglements. The implications of these results are discussed.
- Open Access
- 27 March 2021
- Published by Center for Open Science in PsyArXiv
Machine Learning for the Educational Sciences
Classical statistical methods are limited in the analysis of highdimensional datasets. Machine learning (ML) provides a powerful framework for prediction by using complex relationships, often encountered in modern data with a large number of variables, cases and potentially non-linear effects. ML has turned into one of the most influential analytical approaches of this millennium and has recently become popular in the behavioral and social sciences. The impact of ML methods on research and practical applications in the educational sciences is still limited, but continuously grows as larger and more complex datasets become available through massive open online courses (MOOCs) and large scale investigations.The educational sciences are at a crucial pivot point, because of the anticipated impact ML methods hold for the field. Here, we review the opportunities and challenges of ML for the educational sciences, show how a look at related disciplines can help learning from their experiences, and argue for a philosophical shift in model evaluation. We demonstrate how the overall quality of data analysis in educational research can benefit from these methods and show how ML can play a decisive role in the validation of empirical models. In this review, we (1) provide an overview of the types of data suitable for ML, (2) give practical advice for the application of ML methods, and (3) show how ML-based tools and applications can be used to enhance the quality of education. Additionally we provide practical R code with exemplary analyses, available at https: //osf.io/ntre9/?view only=d29ae7cf59d34e8293f4c6bbde3e4ab2.
- 27 March 2021
- Published by Center for Open Science in PsyArXiv
Open AccessMachine Learning for the Educational Sciences
Classical statistical methods are limited in the analysis of highdimensional datasets. Machine learning (ML) provides a powerful framework for prediction by using complex relationships, often encountered in modern data with a large number of variables, cases and potentially non-linear effects. ML has turned into one of the most influential analytical approaches of this millennium and has recently become popular in the behavioral and social sciences. The impact of ML methods on research and practical applications in the educational sciences is still limited, but continuously grows as larger and more complex datasets become available through massive open online courses (MOOCs) and large scale investigations.The educational sciences are at a crucial pivot point, because of the anticipated impact ML methods hold for the field. Here, we review the opportunities and challenges of ML for the educational sciences, show how a look at related disciplines can help learning from their experiences, and argue for a philosophical shift in model evaluation. We demonstrate how the overall quality of data analysis in educational research can benefit from these methods and show how ML can play a decisive role in the validation of empirical models. In this review, we (1) provide an overview of the types of data suitable for ML, (2) give practical advice for the application of ML methods, and (3) show how ML-based tools and applications can be used to enhance the quality of education. Additionally we provide practical R code with exemplary analyses, available at https: //osf.io/ntre9/?view only=d29ae7cf59d34e8293f4c6bbde3e4ab2.
- Top Cited PapersOpen Access
- 11 December 2019
- Published by The Open Journal in Journal of Open Source Software
- Vol. 4 (44) , 1903
mlr3: A modern object-oriented machine learning framework in R
- 11 December 2019
- Published by The Open Journal in Journal of Open Source Software
- Vol. 4 (44) , 1903
Top Cited PapersOpen Accessmlr3: A modern object-oriented machine learning framework in R