Dougal Hutchison

Background: Education, and information about education, is highly structured: individuals are grouped into classes, which are grouped into schools, which are grouped into local authorities, which are grouped into countries. The degree of similarity among members of a group, such as a school or classroom, is a very important factor in the design and analysis of studies in education. Purpose: The aim of this article is to provide information on this degree of similarity within schools to enable those involved in carrying out surveys of schools to do so most efficiently in terms of resources and minimum disturbance of schools. Sources of data: This paper uses data from 13 studies at primary and secondary level conducted by the National Foundation for Educational Research in England and Wales. Main argument: The degree of similarity among members of a group is measured by two statistics, the intra-cluster correlation and the design effect. The study described here classifies outcomes into a number of categories and estimates intra-cluster correlation and design effect. The relevance of the results to survey design and analysis is discussed, and examples of how to use these are given. Conclusions: The main findings of this study, rather than conclusions as such, are the intra-cluster correlations for each topic category. However, the paper reaches some tentative conclusions about the degree of clustering by topic. Using Hox's convention (Multilevel analysis: Techniques and applications, Lawrence Erlbaum, London, 2002) for the size of intra-cluster correlations, it was found that the degree of clustering of achievement was high, while ethnic and language variables were highly clustered in secondary but not primary. By contrast, attitudes towards school, educationally relevant home characteristics, and perception of school policies have quite low values of ρ (mean < 0.05), defined as small.

There is a degree of instability in any measurement, so that if it is repeated, it is possible that a different result may be obtained. Such instability, generally described as ‘measurement error’, may affect the conclusions drawn from an investigation, and methods exist for allowing it. It is less widely known that different disciplines, and different workers within a discipline, have different interpretations of the term ‘measurement error’, and that these different interpretations are liable to give rise to different results. This article is an attempt to describe and systematise a number of conceptualisations of measurement error, and the associated concept of randomness, and to show how these work out in practice. Because of the importance of this topic, the aim has been to make the descriptions accessible, avoiding jargon and other technical matters such as equations as far as is feasible.

(2008). Concorde and discord: the art of multilevel modelling. International Journal of Research & Method in Education: Vol. 31, No. 1, pp. 11-18.

This paper reports a comparison of human and computer marking of approximately 600 essays produced by 11‐year‐olds in the UK. Each essay script was scored by three human markers. Scripts were also scored by the e‐rater program. There was a good agreement between human and machine marking. Scripts with highly discrepant scores were flagged and assessed blind by expert markers for characteristics considered likely to produce human–machine discrepancies. As hypothesised, essays marked higher by humans exhibited more abstract qualities such as interest and relevance, while there was little, if any, difference on more mechanical factors such as paragraph demarcation.

The Malta Primary Literacy Value‐Added study was the outcome of a process of testing and matching at ages 9–10, the pupils in Maltese primary schools who were tested three years earlier, at ages 6–7. All primary schools in the island took part at both stages. Pupils were tested in both Maltese and English at both ages. In addition extensive data on pupils’ home background was collected at both ages and added to the individual test records. Some 4239 pupils had complete data at both ages. This paper reports the findings of analyses of the resulting data using multilevel modelling. Individualised reports were prepared for each school, taking account of all the background data collected, and a series of one‐to‐one sessions were held with representatives of all schools to discuss how these could be used to help improve the education provided. An anonymised example of a feedback report is given. The authors believe that, in addition to being of interest in its own right, this study can provide a template for other states to carry out a high quality study efficiently and relatively inexpensively, using the special resources of such communities.

The last 20 years have seen the development of sophisticated techniques for analysing individual data within a hierarchical context and the growing availability of good datasets to which these techniques can be applied. The modelling of pupil performance controlling for prior attainment has led to a type of analysis commonly titled ‘value‐added’. Concern for school factors which affect pupil progress has given rise to ‘school effectiveness’ research. This article outlines the history of these linked movements, mainly with reference to England and Wales, but developments in other countries are also outlined. It discusses some of the objections that have been raised, and comments on possible future directions.

This article investigates the existence or otherwise of group‐level effects on progress in reading. ‘Administrative’ data, i.e. data already produced by a local education authority (LEA) for its own purposes, was combined to give two primary‐age cohorts, each of the order of 2500 pupils, in one medium‐sized LEA in the south‐east of England. After allowing for pupil and global school‐level effects, a wide variety of possible aggregated group‐level effects (AGLEs) was investigated. Different functions of pupil score were investigated. Mean score and pupil turnover, especially the latter, were the most important AGLEs on attainment, and these had greater effects for older pupils.