Improving Fairness of Large Language Models in Multi-document Summarization
Preprint
Abstract
Fairness in multi-document summarization (MDS) is crucial for providing comprehensive views across documents with diverse social attribute values, which can significantly impact decision-making. For example, a summarization system that tends to overrepresent negative reviews of products can mislead customers into disregarding good products. Previous works measure fairness in MDS at two levels: summary-level and corpus-level. While summary-level fairness focuses on individual summaries, corpus-level fairness focuses on a corpus of summaries. Recent methods primarily focus on summary-level fairness. We propose FairPO, a preference tuning method that focuses on both summary-level and corpus-level fairness in MDS. To improve summary-level fairness, we propose to generate preference pairs by perturbing document sets. To improve corpus-level fairness, we propose fairness-aware preference tuning by dynamically adjusting the weights of preference pairs. Our experiments show that FairPO outperforms strong baselines while maintaining the critical qualities of summaries. The code is available at https://github.com/leehaoyuan/coverage_fairnes.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Coverage-based Fairness in Multi-document Summarization2025
- Bias in Opinion Summarisation from Pre-training to Adaptation: A Case Study in Political Bias2024
- Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models2024
- Fair Abstractive Summarization of Diverse Perspectives2024
- Polarity Calibration for Opinion Summarization2024
- Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback2023
- Training Language Models to Follow Instructions with Human Feedback2022
- Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects2019
- SemEval-2016 Task 6: Detecting Stance in Tweets2016