Comparative Performance of Two Quantitative Safety Signalling Methods
Authors:
- June S AlmenoffJAJune S Almenoff
- David FramDFDavid Fram
- William DuMouchelWDWilliam DuMouchel
- 1 January 2006
- journal article
- Published by Springer Nature in Drug Safety
- Vol. 29 (10) , 875-887
Abstract
There is increasing interest in using disproportionality-based signal detection methods to support postmarketing safety surveillance activities. Two commonly used methods, empirical Bayes multi-item gamma Poisson shrinker (MGPS) and proportional reporting ratio (PRR), perform differently with respect to the number and types of signals detected. The goal of this study was to compare and analyse the performance characteristics of these two methods, to understand why they differ and to consider the practical implications of these differences for a large, industry-based pharmacovigilance department. We compared the numbers and types of signals of disproportionate reporting (SDRs) obtained with MGPS and PRR using two postmarketing safety databases and a simulated database. We recorded signal counts and performed a qualitative comparison of the drug-event combinations signalled by the two methods as well as a sensitivity analysis to better understand how the thresholds commonly used for these methods impact their performance. PRR detected more SDRs than MGPS. We observed that MGPS is less subject to confounding by demographic factors because it employs stratification and is more stable than PRR when report counts are low. Simulation experiments performed using published empirical thresholds demonstrated that PRR detected false-positive signals at a rate of 1.1%, while MGPS did not detect any statistical false positives. In an attempt to separate the effect of choice of signal threshold from more fundamental methodological differences, we performed a series of experiments in which we modified the conventional threshold values for each method so that each method detected the same number of SDRs for the example drugs studied. This analysis, which provided quantitative examples of the relationship between the published thresholds for the two methods, demonstrates that the signalling criterion published for PRR has a higher signalling frequency than that published for MGPS. The performance differences between the PRR and MGPS methods are related to (i) greater confounding by demographic factors with PRR; (ii) a higher tendency of PRR to detect false-positive signals when the number of reports is small; and (iii) the conventional thresholds that have been adapted for each method. PRR tends to be more 'sensitive' and less 'specific' than MGPS. A high-specificity disproportionality method, when used in conjunction with medical triage and investigation of critical medical events, may provide an efficient and robust approach to applying quantitative methods in routine postmarketing pharmacovigilance.Keywords
This publication has 20 references indexed in Scilit:
- Reply: The evaluation of data mining methods for the simultaneous and systematic detection of safety signals in large databases: lessons to be learnedBritish Journal of Clinical Pharmacology, 2005
- Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance dataStatistics in Medicine, 2005
- A challenge to the data minersPharmacoepidemiology and Drug Safety, 2004
- Drug‐induced pancreatitis: lessons in data miningBritish Journal of Clinical Pharmacology, 2004
- Trimethoprim‐induced hyperkalaemia – lessons in data miningBritish Journal of Clinical Pharmacology, 2004
- Safety Related Drug-Labelling ChangesDrug Safety, 2004
- A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactionsPharmacoepidemiology and Drug Safety, 2002
- Use of Measures of Disproportionality in PharmacovigilanceDrug Safety, 2002
- Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reportsPharmacoepidemiology and Drug Safety, 2001
- A Retrospective Evaluation of a Data Mining Approach to Aid Finding New Adverse Drug Reaction Signals in the WHO International DatabaseDrug Safety, 2000