Improving Fairness using Vision-Language Driven Image Augmentation
Preprint
Abstract
Fairness is crucial when training a deep-learning discriminative model, especially in the facial domain. Models tend to correlate specific characteristics (such as age and skin color) with unrelated attributes (downstream tasks), resulting in biases which do not correspond to reality. It is common knowledge that these correlations are present in the data and are then transferred to the models during training. This paper proposes a method to mitigate these correlations to improve fairness. To do so, we learn interpretable and meaningful paths lying in the semantic space of a pre-trained diffusion model (DiffAE) -- such paths being supervised by contrastive text dipoles. That is, we learn to edit protected characteristics (age and skin color). These paths are then applied to augment images to improve the fairness of a given dataset. We test the proposed method on CelebA-HQ and UTKFace on several downstream tasks with age and skin color as protected characteristics. As a proxy for fairness, we compute the difference in accuracy with respect to the protected characteristics. Quantitative results show how the augmented images help the model improve the overall accuracy, the aforementioned metric, and the disparity of equal opportunity. Code is available at: https://github.com/Moreno98/Vision-Language-Bias-Control.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Unleashing the Transferability Power of Unsupervised Pre-Training for Emotion Recognition in Masked and Unmasked Facial ImagesIEEE Access, 2023
- Challenges in Deploying Machine Learning: A Survey of Case StudiesACM Computing Surveys, 2022
- Face Synthesis With a Focus on Facial Attributes Translation Using Attention MechanismsIEEE Transactions on Biometrics, Behavior, and Identity Science, 2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Tensor Component Analysis for Interpreting the Latent Space of GANs2021
- FairGAN: Fairness-aware Generative Adversarial Networks2018
- Focal Loss for Dense Object DetectionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2018
- Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints2017