Bias Invariant Approaches for Improving Word Embedding Fairness
Open Access
- 21 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1400-1410
Abstract
Many public pre-trained word embeddings have been shown to encode different types of biases. Embeddings are often obtained from training on large pre-existing corpora, and therefore resulting biases can be a reflection of unfair representations in the original data. Bias, in this scenario, is a challenging problem since current mitigation techniques require knowing and understanding existing biases in the embedding, which is not always possible. In this work, we propose to improve word embedding fairness by borrowing methods from the field of data privacy. The idea behind this approach is to treat bias as if it were a special type of training data leakage. This has the unique advantage of not requiring prior knowledge of potential biases in word embeddings. We investigated two types of privacy algorithms, and measured their effect on bias using four different metrics. To investigate techniques from differential privacy, we applied Gaussian perturbation to public pre-trained word embeddings. To investigate noiseless privacy, we applied vector quantization during training. Experiments show that both approaches improve fairness for commonly used embeddings, and additionally, noiseless privacy techniques reduce the size of the resulting embedding representation.Keywords
This publication has 26 references indexed in Scilit:
- Towards Extending Noiseless Privacy2017
- Deep Learning with Differential Privacy2016
- Modeling confounding by half-sibling regressionProceedings of the National Academy of Sciences, 2016
- Matrix Factorization using Window Sampling and Negative Sampling for Improved Word Representations2016
- Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance2016
- Glove: Global Vectors for Word Representation2014
- Noiseless Database Privacy2011
- Introduction to the CoNLL-2003 shared task2003
- Measuring individual differences in implicit cognition: The implicit association test.Journal of Personality and Social Psychology, 1998
- A vector space model for automatic indexingCommunications of the ACM, 1975