Validating hierarchical verbal autopsy expert algorithms in a large data set with known causes of death
Authors:
Open Access
- 20 February 2016
- journal article
- research article
- Published by International Society of Global Health in Journal of Global Health
- Vol. 6 (1) , 010601
Abstract
Physician assessment historically has been the most common method of analyzing verbal autopsy (VA) data. Recently, the World Health Organization endorsed two automated methods, Tariff 2.0 and InterVA-4, which promise greater objectivity and lower cost. A disadvantage of the Tariff method is that it requires a training data set from a prior validation study, while InterVA relies on clinically specified conditional probabilities. We undertook to validate the hierarchical expert algorithm analysis of VA data, an automated, intuitive, deterministic method that does not require a training data set. Using Population Health Metrics Research Consortium study hospital source data, we compared the primary causes of 1629 neonatal and 1456 1-59 month-old child deaths from VA expert algorithms arranged in a hierarchy to their reference standard causes. The expert algorithms were held constant, while five prior and one new "compromise" neonatal hierarchy, and three former child hierarchies were tested. For each comparison, the reference standard data were resampled 1000 times within the range of cause-specific mortality fractions (CSMF) for one of three approximated community scenarios in the 2013 WHO global causes of death, plus one random mortality cause proportions scenario. We utilized CSMF accuracy to assess overall population-level validity, and the absolute difference between VA and reference standard CSMFs to examine particular causes. Chance-corrected concordance (CCC) and Cohen's kappa were used to evaluate individual-level cause assignment. Overall CSMF accuracy for the best-performing expert algorithm hierarchy was 0.80 (range 0.57-0.96) for neonatal deaths and 0.76 (0.50-0.97) for child deaths. Performance for particular causes of death varied, with fairly flat estimated CSMF over a range of reference values for several causes. Performance at the individual diagnosis level was also less favorable than that for overall CSMF (neonatal: best CCC = 0.23, range 0.16-0.33; best kappa = 0.29, 0.23-0.35; child: best CCC = 0.40, 0.19-0.45; best kappa = 0.29, 0.07-0.35). Expert algorithms in a hierarchy offer an accessible, automated method for assigning VA causes of death. Overall population-level accuracy is similar to that of more complex machine learning methods, but without need for a training data set from a prior validation study.Keywords
This publication has 26 references indexed in Scilit:
- Using verbal autopsy to measure causes of death: the comparative performance of existing methodsBMC Medicine, 2014
- Revising the WHO verbal autopsy instrument to facilitate routine cause-of-death monitoringGlobal Health Action, 2013
- Strengthening standardised interpretation of verbal autopsy data: the new InterVA-4 toolGlobal Health Action, 2012
- Global, regional, and national causes of child mortality: an updated systematic analysis for 2010 with time trends since 2000The Lancet, 2012
- Robust metrics for assessing the performance of different verbal autopsy cause assignment methods in validation studiesPopulation Health Metrics, 2011
- Population Health Metrics Research Consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasetsPopulation Health Metrics, 2011
- Global, regional, and national causes of child mortality in 2008: a systematic analysisThe Lancet, 2010
- Verbal autopsy2006
- Validation of Postmortem Interviews to Ascertain Selected Causes of Death in ChildrenInternational Journal of Epidemiology, 1990
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960