Estimating Prediction Qualities without Ground Truth: A Revisit of the Reverse Testing Framework
Authors:
- 1 August 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 49-54
Abstract
To evaluate prediction qualities of machine learning models, it is typically assumed testing samples are labeled. However, testing labels are not always available in practice. A traditional solution is to approximate prediction qualities on testing samples by the qualities on labeled training samples. But this may be limited in that it completely ignores testing samples. In this paper, we present a new approach to estimate prediction qualities on unlabeled testing sample, based on the reverse testing framework [1]. We evaluate the approach with various quality metrics in classification and anomaly detection tasks, and over numerous real-world data sets. Experimental results show the proposed approach gives a more accurate estimate of prediction qualities on testing sample than those on training samples.Keywords
This publication has 5 references indexed in Scilit:
- Reverse Classification Accuracy: Predicting Segmentation Performance in the Absence of Ground TruthIEEE Transactions on Medical Imaging, 2017
- Face Recognition Performance: Role of Demographic InformationIEEE Transactions on Information Forensics and Security, 2012
- Anomaly detectionACM Computing Surveys, 2009
- Reverse testing2006
- A data-driven software tool for enabling cooperative information sharing among police departmentsEuropean Journal of Operational Research, 2002