Frans J. G. Janssens

We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

Objectives  In the scientific literature about reliability, the main approach to increasing reliability seems to involve increasing the number of observers and improving the instrument used. Other aspects for improving reliability – like the training of raters – seem to receive less notice. It is worth asking whether this technical approach could be complemented by training the user of the instrument. A systematic meta‐analytical review of the research literature was performed to answer this question and examine the effectiveness of planned interventions for improving interrater reliability of health care professionals.Method  The databases of PubMed (MEDLINE), Embase, Omega and PsycINFO were searched. The inclusion criteria were met by 57 studies. Details extracted from the studies included the study design, the number of observers and the number of observed cases, the intervention, the type of instrument (whether or not it was highly technical), and statistical information about the agreement before and after the intervention. Interventions were categorized into three groups: training of professionals, improving the diagnostic instrument and a combination of training and improving the instrument. A meta‐analysis was performed by means of linear regression.Results  The interventions were arranged according to their effectiveness in improving the diagnostic instrument (mean change: β = 0.13), training combined with improving the instrument (mean change: β = 0.10) and training (mean change: β = 0.09).Conclusion  On average, although all types of interventions are effective, improving the diagnostic instrument seems to be the most effective. Especially when highly technical instruments were concerned, improvement proved to be very effective (β = 0.52). Because instrumental variables constitute a major source of error, improving the instrument is an important approach. However, this review offers solid arguments that can complement the literature and practice, with a focus on training the user of the instrument.

Reliable and valid judgments are necessary for regulatory authorities to merit confidence from care institutions and society and preserve authority. Moreover, limited reliability and validity of regulatory judgments increase the risk of limited improvement of the quality of health care. The goal of the study is to obtain insight in (dis) advantages of different regulatory instruments for regulation of health care. In this study, the reliability and validity of judgments generated by a lightly structured and highly structured regulatory instrument used by the Dutch Health Care Inspectorate are compared. Results indicate that the lightly structured instrument causes a large variety in discussed topics in regulatory visits: indicators pointing out potential risks in care are not always part of these discussions, by which incentives to improve care remain unjustly undone. Both types of instruments show variations in the meaning of judgments, indicating validity problems. The results of our study suggest that regulation of health care requires thorough appraisal of instruments. Several requirements are identified: first, an instrument that justifies the complexity of care with an accompanying explicit set of standards is necessary. Second, commitment of inspectors to the instrument is essential. And third, training of inspectors is indispensable.

This article demonstrates the value of theoretical ex-ante evaluation of policy programs. We show how a specific policy might not achieve its objectives and illustrate the elements of the policy, which need improvement. These conclusions are based on a reconstruction and evaluation of the theory behind a policy program that aims to increase productivity of schools in the Netherlands, the policy of ‘‘educational governance.’’ We chose this program for its interesting combination of measures on internal quality assurances as well as accountability and supervision. The evaluation clearly shows a flaw in the theory underlying the policy program, which threatens its potential effectiveness. Furthermore, we show that there is a risk of contrary and incompatible interests among actors, as well as some practical reasons why the program might not work. Adjustment of the program is recommended before implementation.

This paper presents an overview of studies into effects and side effects of control mechanisms in education. We focus on effects and side effects of inspection visits and public performance indicators. A first conclusion is that the studies do not provide us with a clear answer to the question of whether inspections have positive causal effects on the quality of schools. Results of studies of publications on public performance indicators are more unambiguous. They lead us to the conclusion that although principals and teachers believe performance indicators are important, parents and pupils take very little notice of these indicators when choosing schools. A third conclusion concerns the occurrence of side effects of school inspections and other control mechanisms in education. Several of the studies discussed clearly refer to the existence of these side effects, such as ‘window dressing’ and other types of ‘gaming’.