A critical note on experimental research methodology in EC
Authors:
- A.E. EibenAEA.E. Eiben
- M. JelasityMJM. Jelasity
- 25 June 2003
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- Vol. 1, 582-587
Abstract
In this paper we point to some essential shortcomings in contemporary practice in performing and documenting experimental re- search in EC. We identify some crucial problems and the limitations of this practice, and elaborate on research directions that should be pursued to improve the quality and relevance of experimental research. I. INTRODUCTION HE aim of this paper is to deliver a contribution to im- proving the commonly practiced experimental research methodology within evolutionary computing. Our main goal is to initiate a discussion about the means and ends of experi- mental EC research. In the long term we hope that this discus- sion will result in a widely accepted and practiced, sound re- search methodology supporting research results that are better founded and more relevant than today. Studying the literature it is clear that there are already some publications that can be related to various aspects of this subject. For the present treat- ment we deliberately chose a "light" citation policy in order to avoid the impression of an annotated bibliography and to em- phasize the messages better, while acknowledging inspiring contributions from fellow researchers. Furthermore, let us ex- plicitly state that much of our criticism on the present practice is applicable to some of our own work too. We approach the methodology question here from the an- gles of generalizability, performance measures, and repro- ducibility. For an illustration of the nature of problems we wish to address here let us consider the following imaginary example of a paper representing common practice in experi- mental EC research. (Note however, that our discussion will not be restricted to this kind of papers.) The imaginary pa- per proposes a new feature in EAs and shows the merits of the new feature by experiments. The experimental section is structured as follows: 1. A testbed of functions, respectively problem instances, is specified.Keywords
This publication has 7 references indexed in Scilit:
- The Wrapper Approach1998
- Empirical Investigation of Multiparent Recombination Operators in Evolution StrategiesEvolutionary Computation, 1997
- No free lunch theorems for optimizationIEEE Transactions on Evolutionary Computation, 1997
- Handbook of Evolutionary Computation1997
- Designing and reporting on computational experiments with heuristic methodsJournal of Heuristics, 1995
- Testing heuristics: We have it all wrongJournal of Heuristics, 1995
- Geometric Algorithms and Combinatorial Optimization1993