Conducting realistic experiments in software engineering
Authors:
- D.I.K. SjobergDSD.I.K. Sjoberg
- B. AndaBAB. Anda
- E. ArisholmEAE. Arisholm
- T. DybaTDT. Dyba
- M. JorgensenMJM. Jorgensen
- A. KarahasanovicAKA. Karahasanovic
- E.F. KorenEKE.F. Koren
- M. VokacMVM. Vokac
- 26 June 2003
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17-26
Abstract
An important goal of most empirical software engineering research is the transfer of research results to industrial applications. Two important obstacles for this transfer are the lack of control of variables of case studies, i.e., the lack of explanatory power, and the lack of realism of controlled experiments. While it may be difficult to increase the explanatory power of case studies, there is a large potential for increasing the realism of controlled software engineering experiments. To convince industry about the validity and applicability of the experimental results, the tasks, subjects and the environments of the experiments should be as realistic as practically possible. Such experiments are, however, more expensive thanexperiments involving students, small tasks and pen-and-paper environments. Consequently, a change towards more realistic experiments requires a change in the amount of resources spent on software engineering experiments.This paper argues that software engineering researchers should apply for resources enabling expensive and realistic software engineering experiments similar to how other researchers apply for resources for expensive software and hardware that are necessary fortheir research. The paper describes experiences from recent experiments that varied in size from involving one software professional for 5 days to 130 software professionals, from 9 consultancy companies, for one day each.Keywords
This publication has 19 references indexed in Scilit:
- The role of experimentation in software engineering: past, current, and future2002
- Visualizing impacts of database schema changes - A controlled experiment2002
- Impact of effort estimates on software project workInformation and Software Technology, 2001
- Quality and Understandability of Use Case Models2001
- A controlled experiment for evaluating quality guidelines on the maintainability of object-oriented designsIEEE Transactions on Software Engineering, 2001
- Towards a framework for empirical assessment of changeability decayJournal of Systems and Software, 2000
- An Empirical Study of Perspective-Based Usability InspectionProceedings of the Human Factors and Ergonomics Society Annual Meeting, 1998
- Should computer scientists experiment more?Computer, 1998
- Experimental models for validating technologyComputer, 1998
- An empirical study of software maintenance tasksJournal of Software Maintenance: Research and Practice, 1995