The validity and reliability of a signal impact assessment tool
- 19 May 2016
- journal article
- research article
- Published by Wiley in Pharmacoepidemiology and Drug Safety
- Vol. 25 (7) , 815-819
Abstract
Background: The Netherlands Pharmacovigilance Centre Lareb uses an operating Signal Impact Assessment Tool (SIAT) as aid in signal selection. SIAT prioritized signals into one of four categories: strong/moderate signal strength and similarly health impact. Although the SIAT has been used for many years, validity and reliability was never explored.Purpose: The aim of this study is to test the validity and reliability of the operating and weight‐adjusted SIAT.Method: For validity testing, judgments of a Delphi panel of three pharmacovigilance experts were used as a ‘gold standard’. First, the panel judged the weighting of the items included in the SIAT. Then, during two phases, the panel rated 40 signals in one of the four categories. Two researchers scored the signals using the SIAT. Panel judgments were compared with scores for the operating and weight‐adjusted SIAT. Inter‐ and intra‐observer variability was also tested. The Cohen's Kappa coefficient (k) was calculated. At least substantial agreement (k > 0.6) was considered to be necessary for an acceptable reliability.Results: Validity did not meet predefined criteria: operating and weight‐adjusted tool, respectively, k‐phase1 = 0.83 and 0.83, k‐phase2 = 0.18 and 0.36. Differences were found for signal strength and health impact. Inter‐ and intra‐observer variabilities were good, k of 0.78 and 0.72, respectively.Conclusions: The SIAT was found to have an insufficient validity and proper reliability. Although SIAT scores should not be decisive in the decision making process, items included can be used as an aid to decide which signals deserve further investigation. Copyright © 2016 John Wiley & Sons, Ltd.Keywords
This publication has 10 references indexed in Scilit:
- Development of a Novel Regulatory Pharmacovigilance Prioritisation System: An Evaluation of Its Performance at the UK Medicines and Healthcare products Regulatory AgencyDrug Safety, 2013
- The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size RequirementsPhysical Therapy, 2005
- Testing and Implementing Signal Impact Analysis in a Regulatory SettingDrug Safety, 2005
- Impact Analysis of Signals Detected from Spontaneous Adverse Drug Reaction Reporting DataDrug Safety, 2005
- Application of Quantitative Signal Detection in the Dutch Spontaneous Reporting System for Adverse Drug ReactionsDrug Safety, 2003
- Signal Selection and Follow-Up in PharmacovigilanceDrug Safety, 2002
- Determinants of signal selection in a spontaneous reporting system for adverse drug reactionsBritish Journal of Clinical Pharmacology, 2001
- Adverse drug reactions: definitions, diagnosis, and managementThe Lancet, 2000
- Responding to drug safety issuesPharmacoepidemiology and Drug Safety, 1999
- The Delphi technique as a forecasting tool: issues and analysisInternational Journal of Forecasting, 1999