Using simulation to assess the sensitivity and specificity of a signal detection tool for multidimensional public health surveillance data
Authors:
- 28 January 2005
- journal article
- research article
- Published by Wiley in Statistics in Medicine
- Vol. 24 (4) , 551-562
Abstract
The objective of the work described in this paper is to develop a means for characterizing the validity of an empirical methodology for detecting signals potentially related to complicated adverse event (AE) coding terms in multidimensional public health surveillance data. The signal detection tool under evaluation is the multi‐item gamma Poisson shrinkage (MGPS) estimation program. We were interested in its potential application to passive surveillance system monitoring, to screen for ‘signals’ of complicated adverse event coding terms (AE terms) in complex and noisy data. The research was to design and produce a flexible and user‐friendly utility for probabilistically defining complicated signals in a database, iterating large numbers of applications of the MGPS detection algorithm and establishing proportions of correct detection events. We sought to establish the specificity of the MGPS by developing a random background using a gradient that ranged from rigorous (but not very relevant) to relevant (but noisy).To establish the sensitivity, signals were defined based on recognized public health issues of interest (such as the introduction of a new vaccine into the population). Methods of representing a signal included a simple pair‐wise association consisting of a new vaccine and one AE term, as well as a more realistic complex of multiple AE terms comprising a ‘syndrome’. A web application has been developed to create and insert signals with user‐defined probabilities in multiple iterations of simulated random background data. Three forms of simulated data based on the vaccine adverse event reporting system (VAERS) cumulative spontaneous database were defined to serve as background noise against which to contrast introduced vaccine adverse event signals: (1) completely random associations between vaccines and AE terms, (2) random associations of vaccine sets and AE term sets preserving naturally observed vaccine co‐occurrences and AE term co‐occurrences and (3) samples from the actual VAERS data as reported. Rates of detection by the MGPS algorithm can be established for specific signal patterns at varying probabilistic intensities in a choice of random background data forms. Knowing these rates is important for determining the degree of response to an MGPS signal detection event in ‘live’ data. Published in 2005 by John Wiley & Sons, Ltd.Keywords
This publication has 15 references indexed in Scilit:
- A Review and Discussion of Prospective Statistical Surveillance in Public HealthJournal of the Royal Statistical Society Series A: Statistics in Society, 2003
- Roundtable on Bioterrorism DetectionJournal of the American Medical Informatics Association, 2002
- Use of Screening Algorithms and Computer Systems to Efficiently Signal Higher-Than-Expected Combinations of Drugs and Events in the US FDA??s Spontaneous Reports DatabaseDrug Safety, 2002
- Signal Selection and Follow-Up in PharmacovigilanceDrug Safety, 2002
- Use of Measures of Disproportionality in PharmacovigilanceDrug Safety, 2002
- Some US Food and Drug Administration perspectives on data mining for pediatric safety AssessmentCurrent Therapeutic Research, 2001
- Data mining in the US Vaccine Adverse Event Reporting System (VAERS): early detection of intussusception and other events after rotavirus vaccinationVaccine, 2001
- The Emerging Science of Very Early Detection of Disease OutbreaksJournal of Public Health Management & Practice, 2001
- A monitoring system for detecting aberrations in public health surveillance reportsStatistics in Medicine, 1999
- A Statistical Algorithm for the Early Detection of Outbreaks of Infectious DiseaseJournal of the Royal Statistical Society Series A: Statistics in Society, 1996