An Event‐Based Approach for Comparing the Performance of Methods for Prospective Medical Product Monitoring
- 4 January 2012
- journal article
- research article
- Published by Wiley in Pharmacoepidemiology and Drug Safety
- Vol. 21 (6) , 631-639
Abstract
Background: Prospective medical product monitoring is intended to alert stakeholders about whether and when safety problems are identifiable in longitudinal electronic healthcare data. Little attention has been given to how to compare methods in this setting.Purpose: To explore aspects of prospective monitoring that should be considered when comparing method performance and to develop a metric that explicitly accounts for these considerations.Methods: We reviewed existing metrics and propose an event‐based approach that classifies exposed outcomes according to whether a prior alert was generated.Results: In comparing performance of methods for prospective monitoring, three factors must be considered: (1) accuracy in alerting; (2) timeliness of alerting; and (3) the trade‐offs between the costs of false negative and false positive alerting. Traditional scenario‐based measures of accuracy, such as sensitivity and specificity, which classify only at the end of monitoring, fail to appreciate timeliness of alerting and impose fixed tradeoffs between false positive versus false negative consequences. We provide an expression that summarizes event‐based sensitivity (the proportion of exposed events that occur after alerting among all exposed events in scenarios with true safety issues) and event‐based specificity (the proportion of exposed events that occur in the absence of alerting among all exposed events in scenarios with no true safety issues) by taking an average weighted by relative costs of false positive and false negative alerting.Conclusions: The proposed approach explicitly accounts for accuracy in alerting, timeliness in alerting, and the trade‐offs between the costs of false negative and false positive alerting. Copyright © 2012 John Wiley & Sons, Ltd.Keywords
Funding Information
- National Institutes of Health (RC1-LM010351, RC1-RR028231, R01-LM010213, RC4-HL102023)
- AHRQ (1 K01 HS018088)
This publication has 13 references indexed in Scilit:
- Automated Surveillance to Detect Postprocedure Safety Signals of Approved Cardiovascular DevicesJAMA, 2010
- Combining electronic healthcare databases in Europe to allow for large‐scale drug safety monitoring: the EU‐ADR ProjectPharmacoepidemiology and Drug Safety, 2010
- Methods for drug safety signal detection in longitudinal observational databases: LGPS and LEOPARDPharmacoepidemiology and Drug Safety, 2010
- A basic study design for expedited safety signal evaluation based on electronic healthcare dataPharmacoepidemiology and Drug Safety, 2010
- Signal detection for vaccine side effects that have not been specified in advancePharmacoepidemiology and Drug Safety, 2010
- Managing Drug-Risk Information — What to Do with All Those New NumbersNew England Journal of Medicine, 2009
- Potential for Conflict of Interest in the Evaluation of Suspected Adverse Drug ReactionsJAMA, 2004
- The diagnostic odds ratio: a single indicator of test performanceJournal of Clinical Epidemiology, 2003
- Precautions for proactive surveillancePharmacoepidemiology and Drug Safety, 2002
- Angio-oedema in relation to treatment with angiotensin converting enzyme inhibitors.BMJ, 1992