Critical Questions Generation: Motivation and Challenges
Preprint
Abstract
The development of Large Language Models (LLMs) has brought impressive performances on mitigation strategies against misinformation, such as counterargument generation. However, LLMs are still seriously hindered by outdated knowledge and by their tendency to generate hallucinated content. In order to circumvent these issues, we propose a new task, namely, Critical Questions Generation, consisting of processing an argumentative text to generate the critical questions (CQs) raised by it. In argumentation theory CQs are tools designed to lay bare the blind spots of an argument by pointing at the information it could be missing. Thus, instead of trying to deploy LLMs to produce knowledgeable and relevant counterarguments, we use them to question arguments, without requiring any external knowledge. Research on CQs Generation using LLMs requires a reference dataset for large scale experimentation. Thus, in this work we investigate two complementary methods to create such a resource: (i) instantiating CQs templates as defined by Walton's argumentation theory and (ii), using LLMs as CQs generators. By doing so, we contribute with a procedure to establish what is a valid CQ and conclude that, while LLMs are reasonable CQ generators, they still have a wide margin for improvement in this task.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation and BeyondIEEE Transactions on Knowledge and Data Engineering, 2025
- NLAS-multi: A multilingual corpus of automatically generated Natural Language Argumentation SchemesData in Brief, 2024
- Scaling up interactive argumentation by providing counterarguments with a chatbotNature Human Behaviour, 2022
- Classifying Argumentative Relations Using Logical Mechanisms and Argumentation SchemesTransactions of the Association for Computational Linguistics, 2021
- Long-term effectiveness of inoculation against misinformation: Three longitudinal experiments.Journal of Experimental Psychology: Applied, 2021
- Let's replay the political debate: Hypervideo technology for visual sensemaking of televised election debatesInternational Journal of Human-Computer Studies, 2021
- Annotating Argument SchemesArgumentation, 2020
- Argumentation in the 2016 US presidential elections: annotated corpora of television debates and social media reactionLanguage Resources and Evaluation, 2019
- Argumentation Schemes2008
- Defeasible reasoningCognitive Science, 1987