Xin He
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Xin He has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Eric. C. Frey
Department of Electrical and Computer Engineering Johns Hopkins University Baltimore MD USA
7 shared publications - Benjamin M. W. Tsui
Department of Radiology, School of Medicine, Johns Hopkins University, Baltimore, MD, USA
2 shared publications B.M.W. Tsui
1 shared publication- Brandon D. Gallas
U.S. Food and Drug Administration (United States)
1 shared publication - Brian Caffo
Department of Biostatistics, Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD 21287, USA
1 shared publication Charles E Metz
1 shared publication
Latest Publications
- 30 October 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 29 (1) , 206-215
Three-Class ROC Analysis—Toward a General Decision Theoretic Solution
Multiclass receiver operating characteristic (ROC) analysis has remained an open theoretical problem since the introduction of binary ROC analysis in the 1950s. Previously, we have developed a paradigm for three-class ROC analysis that extends and unifies decision theoretic, linear discriminant analysis, and probabilistic foundations of binary ROC analysis in a three-class paradigm. One critical element in this paradigm is the equal error utility (EEU) assumption. This assumption allows us to reduce the intrinsic space of the three-class ROC analysis (5-D hypersurface in 6-D hyperspace) to a 2-D surface in the 3-D space of true positive fractions (sensitivity space). In this work, we show that this 2-D ROC surface fully and uniquely provides a complete descriptor for the optimal performance of a system for a three-class classification task, i.e., the triplet of likelihood ratio distributions, assuming such a triplet exists. To be specific, we consider two classifiers that utilize likelihood ratios, and we assumed each classifier has a continuous and differentiable 2-D sensitivity-space ROC surface. Under these conditions, we proved that the classifiers have the same triplet of likelihood ratio distributions if and only if they have the same 2-D sensitivity-space ROC surfaces. As a result, the 2-D sensitivity surface contains complete information on the optimal three-class task performance for the corresponding likelihood ratio classifier.
- 30 October 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 29 (1) , 206-215
Three-Class ROC Analysis—Toward a General Decision Theoretic Solution
Multiclass receiver operating characteristic (ROC) analysis has remained an open theoretical problem since the introduction of binary ROC analysis in the 1950s. Previously, we have developed a paradigm for three-class ROC analysis that extends and unifies decision theoretic, linear discriminant analysis, and probabilistic foundations of binary ROC analysis in a three-class paradigm. One critical element in this paradigm is the equal error utility (EEU) assumption. This assumption allows us to reduce the intrinsic space of the three-class ROC analysis (5-D hypersurface in 6-D hyperspace) to a 2-D surface in the 3-D space of true positive fractions (sensitivity space). In this work, we show that this 2-D ROC surface fully and uniquely provides a complete descriptor for the optimal performance of a system for a three-class classification task, i.e., the triplet of likelihood ratio distributions, assuming such a triplet exists. To be specific, we consider two classifiers that utilize likelihood ratios, and we assumed each classifier has a continuous and differentiable 2-D sensitivity-space ROC surface. Under these conditions, we proved that the classifiers have the same triplet of likelihood ratio distributions if and only if they have the same 2-D sensitivity-space ROC surfaces. As a result, the 2-D sensitivity surface contains complete information on the optimal three-class task performance for the corresponding likelihood ratio classifier.
- 30 September 2009
- Published by Elsevier in Journal of the American College of Radiology
- Vol. 6 (9) , 652-655
ROC, LROC, FROC, AFROC: An Alphabet Soup
- 30 September 2009
- Published by Elsevier in Journal of the American College of Radiology
- Vol. 6 (9) , 652-655
ROC, LROC, FROC, AFROC: An Alphabet Soup
- 16 June 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Nuclear Science
- Vol. 56 (3) , 671-676
Evaluation of Respiratory Motion Effect on Defect Detection in Myocardial Perfusion SPECT: A Simulation Study
The objective of this study is to investigate the effects of respiratory motion (RM) on defect detection in Tc-99m sestamibi myocardial perfusion SPECT (MPS) using a phantom population that includes patient variability. Three RM patterns are included, namely breath-hold, slightly enhanced normal breathing, and deep breathing. For each RM pattern, six 4-D NCAT phantoms were generated, each with anatomical variations. Anterior, lateral and inferior myocardial defects with different sizes and contrasts were inserted. Noise-free SPECT projections were simulated using an analytical projector. Poisson noise was then added to generate noisy realizations. The projection data were reconstructed using the OS-EM algorithm with 1 and 4 subsets/iteration and at 1, 2, 3, 5, 7, and 10 iterations. Short-axis images centered at the centroid of the myocardial defect were extracted, and the channelized Hotelling observer (CHO) was applied for the detection of the defect. The CHO results show that the value of the area under the receiver operating characteristics (ROC) curve (AUC) is affected by the RM amplitude. For all the defect sizes and contrasts studied, the highest or optimal AUC values indicate maximum detectability decrease with the increase of the RM amplitude. With no respiration, the ranking of the optimal AUC value in decreasing order is anterior then lateral, and finally inferior defects. The AUC value of the lateral defect drops more severely as the RM amplitude increases compared to other defect locations. Furthermore, as the RM amplitude increases, the AUC values of the smaller defects drop more quickly than the larger ones. We demonstrated that RM affects defect detectability of MPS imaging. The results indicate that developments of optimal data acquisition methods and RM correction methods are needed to improve the defect detectability in MPS.
- 16 June 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Nuclear Science
- Vol. 56 (3) , 671-676
Evaluation of Respiratory Motion Effect on Defect Detection in Myocardial Perfusion SPECT: A Simulation Study
The objective of this study is to investigate the effects of respiratory motion (RM) on defect detection in Tc-99m sestamibi myocardial perfusion SPECT (MPS) using a phantom population that includes patient variability. Three RM patterns are included, namely breath-hold, slightly enhanced normal breathing, and deep breathing. For each RM pattern, six 4-D NCAT phantoms were generated, each with anatomical variations. Anterior, lateral and inferior myocardial defects with different sizes and contrasts were inserted. Noise-free SPECT projections were simulated using an analytical projector. Poisson noise was then added to generate noisy realizations. The projection data were reconstructed using the OS-EM algorithm with 1 and 4 subsets/iteration and at 1, 2, 3, 5, 7, and 10 iterations. Short-axis images centered at the centroid of the myocardial defect were extracted, and the channelized Hotelling observer (CHO) was applied for the detection of the defect. The CHO results show that the value of the area under the receiver operating characteristics (ROC) curve (AUC) is affected by the RM amplitude. For all the defect sizes and contrasts studied, the highest or optimal AUC values indicate maximum detectability decrease with the increase of the RM amplitude. With no respiration, the ranking of the optimal AUC value in decreasing order is anterior then lateral, and finally inferior defects. The AUC value of the lateral defect drops more severely as the RM amplitude increases compared to other defect locations. Furthermore, as the RM amplitude increases, the AUC values of the smaller defects drop more quickly than the larger ones. We demonstrated that RM affects defect detectability of MPS imaging. The results indicate that developments of optimal data acquisition methods and RM correction methods are needed to improve the defect detectability in MPS.
- 29 July 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (11) , 1556-1567
Application of Three-Class ROC Analysis to Task-Based Image Quality Assessment of Simultaneous Dual-Isotope Myocardial Perfusion SPECT (MPS)
The diagnosis of cardiac disease using dual-isotope myocardial perfusion SPECT (MPS) is based on the defect status in both stress and rest images, and can be modeled as a three-class task of classifying patients as having no, reversible, or fixed perfusion defects. Simultaneous acquisition protocols for dual-isotope MPS imaging have gained much interest due to their advantages including perfect registration of the 201Tl and 99mTc images in space and time, increased patient comfort, and higher clinical throughput. As a result of simultaneous acquisition, however, crosstalk contamination, where photons emitted by one isotope contribute to the image of the other isotope, degrades image quality. Minimizing the crosstalk is important in obtaining the best possible image quality. One way to minimize the crosstalk is to optimize the injected activity of the two isotopes by considering the three-class nature of the diagnostic problem. To effectively do so, we have previously developed a three-class receiver operating characteristic (ROC) analysis methodology that extends and unifies the decision theoretic, linear discriminant analysis, and psychophysical foundations of binary ROC analysis in a three-class paradigm. In this work, we applied the proposed three-class ROC methodology to the assessment of the image quality of simultaneous dual-isotope MPS imaging techniques and the determination of the optimal injected activity combination. In addition to this application, the rapid development of diagnostic imaging techniques has produced an increasing number of clinical diagnostic tasks that involve not only disease detection, but also disease characterization and are thus multiclass tasks. This paper provides a practical example of the application of the proposed three-class ROC analysis methodology to medical problems.
- 29 July 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (11) , 1556-1567
Application of Three-Class ROC Analysis to Task-Based Image Quality Assessment of Simultaneous Dual-Isotope Myocardial Perfusion SPECT (MPS)
The diagnosis of cardiac disease using dual-isotope myocardial perfusion SPECT (MPS) is based on the defect status in both stress and rest images, and can be modeled as a three-class task of classifying patients as having no, reversible, or fixed perfusion defects. Simultaneous acquisition protocols for dual-isotope MPS imaging have gained much interest due to their advantages including perfect registration of the 201Tl and 99mTc images in space and time, increased patient comfort, and higher clinical throughput. As a result of simultaneous acquisition, however, crosstalk contamination, where photons emitted by one isotope contribute to the image of the other isotope, degrades image quality. Minimizing the crosstalk is important in obtaining the best possible image quality. One way to minimize the crosstalk is to optimize the injected activity of the two isotopes by considering the three-class nature of the diagnostic problem. To effectively do so, we have previously developed a three-class receiver operating characteristic (ROC) analysis methodology that extends and unifies the decision theoretic, linear discriminant analysis, and psychophysical foundations of binary ROC analysis in a three-class paradigm. In this work, we applied the proposed three-class ROC methodology to the assessment of the image quality of simultaneous dual-isotope MPS imaging techniques and the determination of the optimal injected activity combination. In addition to this application, the rapid development of diagnostic imaging techniques has produced an increasing number of clinical diagnostic tasks that involve not only disease detection, but also disease characterization and are thus multiclass tasks. This paper provides a practical example of the application of the proposed three-class ROC analysis methodology to medical problems.
- 29 July 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 28 (2) , 185-193
The Validity of Three-Class Hotelling Trace (3-HT) in Describing Three-Class Task Performance: Comparison of Three-Class Volume Under ROC Surface (VUS) and 3-HT
In order to describe multiclass classification performance, several figures of merit (FOM) have been proposed. Among the earliest and most widely known of these is the three-class Hotelling trace (3-HT). The goal of this paper is to present theoretical and empirical data demonstrating the failure of 3-HT as a measure of three-class task performance. To help do this, we contrast it to a newly proposed three-class FOM, the volume under the three-class receiver operating characteristic (ROC) surface (VUS). The VUS is obtained from a decision theory based three-class ROC analysis method which has been proved to extend the decision theoretic, linear discriminant analysis (LDA), and psychophysical foundations of binary ROC analysis to a three-class paradigm. We demonstrate empirically that the VUS and 3-HT do not have a monotonic relationship in general when describing three-class task performance. Numerical experiments demonstrated that the VUS provided reasonable results, while the 3-HT failed to distinguish between the case where all objects could be perfectly classified from the case where only one pair of the classes could be perfectly classified. We have provided theoretical explanations of this failure of 3-HT. The significance of this work goes beyond merely demonstrating the problems of the 3-HT, it demonstrates that a FOM that is mathematically correct and has a strong theoretical basis can provide results that violate a common sense understanding of three-class task performance. This fact raises the question of ldquohow to evaluate a classification performance evaluation method?rdquo We believe the answer to this question lies in the theoretical foundations of binary ROC analysis. We have thus contrasted the two FOMs in terms of three fundamental theories underlying binary ROC analysis: decision theory, binary linear discriminant analysis, and the equivalence of two psychophysical classification procedures. These theoretical investigations demonstrated the importance of extending and unifying all the fundamental theories of binary classification in the development of a three-class FOM; violating one of these fundamental binary classification theories may, as it did for the L-HT, provide predictions of three-class task performance that do not agree with a common sense understanding of three-class task performance.
- 29 July 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 28 (2) , 185-193
The Validity of Three-Class Hotelling Trace (3-HT) in Describing Three-Class Task Performance: Comparison of Three-Class Volume Under ROC Surface (VUS) and 3-HT
In order to describe multiclass classification performance, several figures of merit (FOM) have been proposed. Among the earliest and most widely known of these is the three-class Hotelling trace (3-HT). The goal of this paper is to present theoretical and empirical data demonstrating the failure of 3-HT as a measure of three-class task performance. To help do this, we contrast it to a newly proposed three-class FOM, the volume under the three-class receiver operating characteristic (ROC) surface (VUS). The VUS is obtained from a decision theory based three-class ROC analysis method which has been proved to extend the decision theoretic, linear discriminant analysis (LDA), and psychophysical foundations of binary ROC analysis to a three-class paradigm. We demonstrate empirically that the VUS and 3-HT do not have a monotonic relationship in general when describing three-class task performance. Numerical experiments demonstrated that the VUS provided reasonable results, while the 3-HT failed to distinguish between the case where all objects could be perfectly classified from the case where only one pair of the classes could be perfectly classified. We have provided theoretical explanations of this failure of 3-HT. The significance of this work goes beyond merely demonstrating the problems of the 3-HT, it demonstrates that a FOM that is mathematically correct and has a strong theoretical basis can provide results that violate a common sense understanding of three-class task performance. This fact raises the question of ldquohow to evaluate a classification performance evaluation method?rdquo We believe the answer to this question lies in the theoretical foundations of binary ROC analysis. We have thus contrasted the two FOMs in terms of three fundamental theories underlying binary ROC analysis: decision theory, binary linear discriminant analysis, and the equivalence of two psychophysical classification procedures. These theoretical investigations demonstrated the importance of extending and unifying all the fundamental theories of binary classification in the development of a three-class FOM; violating one of these fundamental binary classification theories may, as it did for the L-HT, provide predictions of three-class task performance that do not agree with a common sense understanding of three-class task performance.
- 2 May 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (10) , 1535-1543
Toward Realistic and Practical Ideal Observer (IO) Estimation for the Optimization of Medical Imaging Systems
The ideal observer (IO) employs complete knowledge of the available data statistics and sets an upper limit on observer performance on a binary classification task. However, the IO test statistic cannot be calculated analytically, except for cases where object statistics are extremely simple. Kupinski et al. have developed a Markov chain Monte Carlo (MCMC) based technique to compute the IO test statistic for, in principle, arbitrarily complex objects and imaging systems. In this work, we applied MCMC to estimate the IO test statistic in the context of myocardial perfusion SPECT (MPS). We modeled the imaging system using an analytic SPECT projector with attenuation, distant-dependent detector-response modeling and Poisson noise statistics. The object is a family of parameterized torso phantoms with variable geometric and organ uptake parameters. To accelerate the imaging simulation process and thus enable the MCMC IO estimation, we used discretized anatomic parameters and continuous uptake parameters in defining the objects. The imaging process simulation was modeled by precomputing projections for each organ for a finite number of discretely-parameterized anatomic parameters and taking linear combinations of the organ projections based on continuous sampling of the organ uptake parameters. The proposed method greatly reduces the computational burden and allows MCMC IO estimation for a realistic MPS imaging simulation. We validated the proposed IO estimation technique by estimating IO test statistics for a large number of input objects. The properties of the first- and second-order statistics of the IO test statistics estimated using the MCMC IO estimation technique agreed well with theoretical predictions. Further, as expected, the IO had better performance, as measured by the receiver operating characteristic (ROC) curve, than the Hotelling observer. This method is developed for SPECT imaging. However, it can be adapted to any linear imaging system.
- 2 May 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (10) , 1535-1543
Toward Realistic and Practical Ideal Observer (IO) Estimation for the Optimization of Medical Imaging Systems
The ideal observer (IO) employs complete knowledge of the available data statistics and sets an upper limit on observer performance on a binary classification task. However, the IO test statistic cannot be calculated analytically, except for cases where object statistics are extremely simple. Kupinski et al. have developed a Markov chain Monte Carlo (MCMC) based technique to compute the IO test statistic for, in principle, arbitrarily complex objects and imaging systems. In this work, we applied MCMC to estimate the IO test statistic in the context of myocardial perfusion SPECT (MPS). We modeled the imaging system using an analytic SPECT projector with attenuation, distant-dependent detector-response modeling and Poisson noise statistics. The object is a family of parameterized torso phantoms with variable geometric and organ uptake parameters. To accelerate the imaging simulation process and thus enable the MCMC IO estimation, we used discretized anatomic parameters and continuous uptake parameters in defining the objects. The imaging process simulation was modeled by precomputing projections for each organ for a finite number of discretely-parameterized anatomic parameters and taking linear combinations of the organ projections based on continuous sampling of the organ uptake parameters. The proposed method greatly reduces the computational burden and allows MCMC IO estimation for a realistic MPS imaging simulation. We validated the proposed IO estimation technique by estimating IO test statistics for a large number of input objects. The properties of the first- and second-order statistics of the IO test statistics estimated using the MCMC IO estimation technique agreed well with theoretical predictions. Further, as expected, the IO had better performance, as measured by the receiver operating characteristic (ROC) curve, than the Hotelling observer. This method is developed for SPECT imaging. However, it can be adapted to any linear imaging system.
- 25 April 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (5) , 577-588
The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS)
Previously, we have proposed a method for three-class receiver operating characteristic (ROC) analysis based on decision theory. In this method, the volume under a three-class ROC surface (VUS) serves as a figure-of-merit (FOM) and measures three-class task performance. The proposed three-class ROC analysis method was demonstrated to be optimal under decision theory according to several decision criteria. Further, an optimal three-class linear observer was proposed to simultaneously maximize the signal-to-noise ratio (SNR) between the test statistics of each pair of the classes provided certain data linearity condition. Applicability of this three-class ROC analysis method would be further enhanced by the development of an intuitive meaning of the VUS and a more general method to calculate the VUS that provides an estimate of its standard error. In this paper, we investigated the general meaning and usage of VUS as a FOM for three-class classification task performance. We showed that the VUS value, which is obtained from a rating procedure, equals the percent correct in a corresponding categorization procedure for continuous rating data. The significance of this relationship goes beyond providing another theoretical basis for three-class ROC analysis - it enables statistical analysis of the VUS value. Based on this relationship, we developed and tested algorithms for calculating the VUS and its variance. Finally, we reviewed the current status of the proposed three-class ROC analysis methodology, and concluded that it extends and unifies decision theoretic, linear discriminant analysis, and psychophysical foundations of binary ROC analysis in a three-class paradigm.
- 25 April 2008
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 27 (5) , 577-588
The Meaning and Use of the Volume Under a Three-Class ROC Surface (VUS)
Previously, we have proposed a method for three-class receiver operating characteristic (ROC) analysis based on decision theory. In this method, the volume under a three-class ROC surface (VUS) serves as a figure-of-merit (FOM) and measures three-class task performance. The proposed three-class ROC analysis method was demonstrated to be optimal under decision theory according to several decision criteria. Further, an optimal three-class linear observer was proposed to simultaneously maximize the signal-to-noise ratio (SNR) between the test statistics of each pair of the classes provided certain data linearity condition. Applicability of this three-class ROC analysis method would be further enhanced by the development of an intuitive meaning of the VUS and a more general method to calculate the VUS that provides an estimate of its standard error. In this paper, we investigated the general meaning and usage of VUS as a FOM for three-class classification task performance. We showed that the VUS value, which is obtained from a rating procedure, equals the percent correct in a corresponding categorization procedure for continuous rating data. The significance of this relationship goes beyond providing another theoretical basis for three-class ROC analysis - it enables statistical analysis of the VUS value. Based on this relationship, we developed and tested algorithms for calculating the VUS and its variance. Finally, we reviewed the current status of the proposed three-class ROC analysis methodology, and concluded that it extends and unifies decision theoretic, linear discriminant analysis, and psychophysical foundations of binary ROC analysis in a three-class paradigm.
- 26 December 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 26 (1) , 77-83
An Optimal Three-Class Linear Observer Derived From Decision Theory
Many attempts have been made to develop an optimal linear observer for classifying multiclass data. Most approaches either do not have a definite description of optimality or have regions of ambiguity in decision making. In this paper, we derive a three-class Hotelling observer (3-HO), inspired by the ideal observer that results from a decision theoretic solution to the three-class classification problem. Assuming the data vectors follow multivariate Gaussian distributions with equal covariance matrices for the three classes, it is shown that two two-class Hotelling templates construct a 3-HO which has the same performance as the three-class ideal observer (3-IO). We show that, without the Gaussian and equal covariance assumptions, the 3-HO is still applicable when the two-class Hotelling templates of each pair of the classes satisfy a certain linear relationship. In this case, the 3-HO simultaneously maximizes the signal-to-noise (SNR) of the test statistics between each pair of the classes. In conclusion, we developed a three-class linear mathematical observer that uses first- and second-order ensemble data statistics. This mathematical observer, which has clearly defined optimality for several data statistics conditions and has decision rules that have no ambiguous decision regions, is potentially useful in the optimization and evaluation of imaging techniques for performing three-class diagnostic tasks
- 26 December 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 26 (1) , 77-83
An Optimal Three-Class Linear Observer Derived From Decision Theory
Many attempts have been made to develop an optimal linear observer for classifying multiclass data. Most approaches either do not have a definite description of optimality or have regions of ambiguity in decision making. In this paper, we derive a three-class Hotelling observer (3-HO), inspired by the ideal observer that results from a decision theoretic solution to the three-class classification problem. Assuming the data vectors follow multivariate Gaussian distributions with equal covariance matrices for the three classes, it is shown that two two-class Hotelling templates construct a 3-HO which has the same performance as the three-class ideal observer (3-IO). We show that, without the Gaussian and equal covariance assumptions, the 3-HO is still applicable when the two-class Hotelling templates of each pair of the classes satisfy a certain linear relationship. In this case, the 3-HO simultaneously maximizes the signal-to-noise (SNR) of the test statistics between each pair of the classes. In conclusion, we developed a three-class linear mathematical observer that uses first- and second-order ensemble data statistics. This mathematical observer, which has clearly defined optimality for several data statistics conditions and has decision rules that have no ambiguous decision regions, is potentially useful in the optimization and evaluation of imaging techniques for performing three-class diagnostic tasks
- 1 May 2006
- Vol. 25 (5)
Three-class ROC analysis--a decision theoretic approach under the ideal observer framework.
- 1 May 2006
- Vol. 25 (5)
Three-class ROC analysis--a decision theoretic approach under the ideal observer framework.
- 1 May 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 25 (8) , 979-986
Three-Class ROC Analysis—The Equal Error Utility Assumption and the Optimality of Three-Class ROC Surface Using the Ideal Observer
Receiver operating characteristic (ROC) analysis is well established in the evaluation of systems involving binary classification tasks. However, medical tests often require distinguishing among more than two diagnostic alternatives. The goal of this work was to develop an ROC analysis method for three-class classification tasks. Based on decision theory, we developed a method for three-class ROC analysis. In this method, the objects were classified by making the decision that provided the maximal utility relative to the other two. By making assumptions about the magnitudes of the relative utilities of incorrect decisions, we found a decision model that maximized the expected utility of the decisions when using log-likelihood ratios as decision variables. This decision model consists of a two-dimensional decision plane with log likelihood ratios as the axes and a decision structure that separates the plane into three regions. Moving the decision structure over the decision plane, which corresponds to moving the decision threshold in two-class ROC analysis, and computing the true class 1, 2, and 3 fractions defined a three-class ROC surface. We have shown that the resulting three-class ROC surface shares many features with the two-class ROC curve; i.e., using the log likelihood ratios as the decision variables results in maximal expected utility of the decisions, and the optimal operating point for a given diagnostic setting (set of relative utilities and disease prevalences) lies on the surface. The volume under the three-class surface (VUS) serves as a figure-of-merit to evaluate different data acquisition systems or image processing and reconstruction methods when the assumed utility constraints are relevant.
- 1 May 2006
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Medical Imaging
- Vol. 25 (8) , 979-986
Three-Class ROC Analysis—The Equal Error Utility Assumption and the Optimality of Three-Class ROC Surface Using the Ideal Observer
Receiver operating characteristic (ROC) analysis is well established in the evaluation of systems involving binary classification tasks. However, medical tests often require distinguishing among more than two diagnostic alternatives. The goal of this work was to develop an ROC analysis method for three-class classification tasks. Based on decision theory, we developed a method for three-class ROC analysis. In this method, the objects were classified by making the decision that provided the maximal utility relative to the other two. By making assumptions about the magnitudes of the relative utilities of incorrect decisions, we found a decision model that maximized the expected utility of the decisions when using log-likelihood ratios as decision variables. This decision model consists of a two-dimensional decision plane with log likelihood ratios as the axes and a decision structure that separates the plane into three regions. Moving the decision structure over the decision plane, which corresponds to moving the decision threshold in two-class ROC analysis, and computing the true class 1, 2, and 3 fractions defined a three-class ROC surface. We have shown that the resulting three-class ROC surface shares many features with the two-class ROC curve; i.e., using the log likelihood ratios as the decision variables results in maximal expected utility of the decisions, and the optimal operating point for a given diagnostic setting (set of relative utilities and disease prevalences) lies on the surface. The volume under the three-class surface (VUS) serves as a figure-of-merit to evaluate different data acquisition systems or image processing and reconstruction methods when the assumed utility constraints are relevant.