Conformal Correction for Efficiency May be at Odds with Entropy
Preprint
Abstract
Conformal prediction (CP) provides a comprehensive framework to produce statistically rigorous uncertainty sets for black-box machine learning models. To further improve the efficiency of CP, conformal correction is proposed to fine-tune or wrap the base model with an extra module using a conformal-aware inefficiency loss. In this work, we empirically and theoretically identify a trade-off between the CP efficiency and the entropy of model prediction. We then propose an entropy-constrained conformal correction method, exploring a better Pareto optimum between efficiency and entropy. Extensive experimental results on both computer vision and graph datasets demonstrate the efficacy of the proposed method. For instance, it can significantly improve the efficiency of state-of-the-art CP methods by up to 34.4%, given an entropy threshold.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- Conformal prediction: A unified review of theory and new challengesBernoulli, 2023
- Uncertainty Quantification over Graph with Conformalized Graph Neural Networks2023
- On the Utility of Prediction Sets in Human-AI Teams2022
- Training Uncertainty-Aware Classifiers with Conformalized Deep Learning2022
- Algorithmic Learning in a Random World2022
- A review of uncertainty quantification in deep learning: Techniques, applications and challengesInformation Fusion, 2021
- Least Ambiguous Set-Valued Classifiers With Bounded Error LevelsJournal of the American Statistical Association, 2018
- A conformal prediction approach to explore functional dataAnnals of Mathematics and Artificial Intelligence, 2013
- Cross-conformal predictorsAnnals of Mathematics and Artificial Intelligence, 2013
- Automating the Construction of Internet Portals with Machine LearningInformation Retrieval Journal, 2000