Abstract
In this paper we point to some essential shortcomings in contemporary practice in performing and documenting experimental re- search in EC. We identify some crucial problems and the limitations of this practice, and elaborate on research directions that should be pursued to improve the quality and relevance of experimental research. I. INTRODUCTION HE aim of this paper is to deliver a contribution to im- proving the commonly practiced experimental research methodology within evolutionary computing. Our main goal is to initiate a discussion about the means and ends of experi- mental EC research. In the long term we hope that this discus- sion will result in a widely accepted and practiced, sound re- search methodology supporting research results that are better founded and more relevant than today. Studying the literature it is clear that there are already some publications that can be related to various aspects of this subject. For the present treat- ment we deliberately chose a "light" citation policy in order to avoid the impression of an annotated bibliography and to em- phasize the messages better, while acknowledging inspiring contributions from fellow researchers. Furthermore, let us ex- plicitly state that much of our criticism on the present practice is applicable to some of our own work too. We approach the methodology question here from the an- gles of generalizability, performance measures, and repro- ducibility. For an illustration of the nature of problems we wish to address here let us consider the following imaginary example of a paper representing common practice in experi- mental EC research. (Note however, that our discussion will not be restricted to this kind of papers.) The imaginary pa- per proposes a new feature in EAs and shows the merits of the new feature by experiments. The experimental section is structured as follows: 1. A testbed of functions, respectively problem instances, is specified.