Comprehensive Equity Index (CEI): Definition and Application to Bias Evaluation in Biometrics
Preprint
Abstract
We present a novel metric designed, among other applications, to quantify biased behaviors of machine learning models. As its core, the metric consists of a new similarity metric between score distributions that balances both their general shapes and tails' probabilities. In that sense, our proposed metric may be useful in many application areas. Here we focus on and apply it to the operational evaluation of face recognition systems, with special attention to quantifying demographic biases; an application where our metric is especially useful. The topic of demographic bias and fairness in biometric recognition systems has gained major attention in recent years. The usage of these systems has spread in society, raising concerns about the extent to which these systems treat different population groups. A relevant step to prevent and mitigate demographic biases is first to detect and quantify them. Traditionally, two approaches have been studied to quantify differences between population groups in machine learning literature: 1) measuring differences in error rates, and 2) measuring differences in recognition score distributions. Our proposed Comprehensive Equity Index (CEI) trade-offs both approaches combining both errors from distribution tails and general distribution shapes. This new metric is well suited to real-world scenarios, as measured on NIST FRVT evaluations, involving high-performance systems and realistic face databases including a wide range of covariates and demographic groups. We first show the limitations of existing metrics to correctly assess the presence of biases in realistic setups and then propose our new metric to tackle these limitations. We tested the proposed metric with two state-of-the-art models and four widely used databases, showing its capacity to overcome the main flaws of previous bias metrics.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- How Good Is ChatGPT at Face Biometrics? A First Look Into Recognition, Soft Biometrics, and ExplainabilityIEEE Access, 2024
- SwipeFormer: Transformers for mobile touchscreen biometricsExpert Systems with Applications, 2023
- Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulationInformation Fusion, 2023
- Human-Centric Multimodal Machine Learning: Recent Advances and Testbed on AI-Based RecruitmentSN Computer Science, 2023
- Facilitating free travel in the Schengen area—A position paper by the European Association for BiometricsIET Biometrics, 2023
- Sensitive loss: Improving accuracy and fairness of face representations with discrimination-aware deep learningArtificial Intelligence, 2022
- Deep face recognition: A surveyNeurocomputing, 2021
- Super-resolution for Selfie Biometrics: Introduction and Application to Face and Iris2019
- The Effect of Broad and Specific Demographic Homogeneity on the Imposter Distributions and False Match Rates in Face Recognition Algorithm Performance2019
- Identification using face regions: Application and assessment in forensic scenariosForensic Science International, 2013