Towards Understanding Gender Bias in Relation Extraction
Preprint
Abstract
Recent developments in Neural Relation Extraction (NRE) have made significant strides towards Automated Knowledge Base Construction (AKBC). While much attention has been dedicated towards improvements in accuracy, there have been no attempts in the literature to our knowledge to evaluate social biases in NRE systems. We create WikiGenderBias, a distantly supervised dataset with a human annotated test set. WikiGenderBias has sentences specifically curated to analyze gender bias in relation extraction systems. We use WikiGenderBias to evaluate systems for bias and find that NRE systems exhibit gender biased predictions and lay groundwork for future evaluation of bias in NRE. We also analyze how name anonymization, hard debiasing for word embeddings, and counterfactual data augmentation affect gender bias in predictions and performance.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Improving Neural Relation Extraction with Implicit Mutual Relations2020
- The Woman Worked as a Babysitter: On Biases in Language Generation2019
- OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction2019
- It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution2019
- Word embeddings quantify 100 years of gender and ethnic stereotypesProceedings of the National Academy of Sciences of the United States of America, 2018
- RESIDE: Improving Distantly-Supervised Neural Relation Extraction using Side Information2018
- Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods2018
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017
- Open information extraction from the webCommunications of the ACM, 2008
- Unsupervised named-entity extraction from the Web: An experimental studyArtificial Intelligence, 2005