Intent-conditioned and Non-toxic Counterspeech Generation using Multi-Task Instruction Tuning with RLAIF
Preprint
Abstract
Counterspeech, defined as a response to mitigate online hate speech, is increasingly used as a non-censorial solution. Addressing hate speech effectively involves dispelling the stereotypes, prejudices, and biases often subtly implied in brief, single-sentence statements or abuses. These implicit expressions challenge language models, especially in seq2seq tasks, as model performance typically excels with longer contexts. Our study introduces CoARL, a novel framework enhancing counterspeech generation by modeling the pragmatic implications underlying social biases in hateful statements. CoARL's first two phases involve sequential multi-instruction tuning, teaching the model to understand intents, reactions, and harms of offensive statements, and then learning task-specific low-rank adapter weights for generating intent-conditioned counterspeech. The final phase uses reinforcement learning to fine-tune outputs for effectiveness and non-toxicity. CoARL outperforms existing benchmarks in intent-conditioned counterspeech generation, showing an average improvement of 3 points in intent-conformity and 4 points in argument-quality metrics. Extensive human evaluation supports CoARL's efficacy in generating superior and more context-appropriate responses compared to existing systems, including prominent LLMs like ChatGPT.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Handling Bias in Toxic Speech Detection: A SurveyACM Computing Surveys, 2023
- Is hate speech detection the solution the world wants?Proceedings of the National Academy of Sciences, 2023
- Empathy-based counterspeech can reduce racist hate speech in a social media field experimentProceedings of the National Academy of Sciences, 2021
- Project Debater APIs: Decomposing the AI Grand Challenge2021
- A Large-Scale Dataset for Argument Quality Ranking: Construction and AnalysisProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Multilingual Argument Mining: Datasets and Analysis2020
- Social Bias Frames: Reasoning about Social and Power Implications of Language2020
- Complex Program Induction for Querying Knowledge Bases in the Absence of Gold ProgramsTransactions of the Association for Computational Linguistics, 2019
- Deep Reinforcement Learning for Sequence-to-Sequence ModelsIEEE Transactions on Neural Networks and Learning Systems, 2019
- Emotional, behavioral, and cognitive reactions to microaggressions: Transgender perspectives.Psychology of Sexual Orientation and Gender Diversity, 2014