Can large language models generate salient negative statements?
Preprint
Abstract
We examine the ability of large language models (LLMs) to generate salient (interesting) negative statements about real-world entities; an emerging research topic of the last few years. We probe the LLMs using zero- and k-shot unconstrained probes, and compare with traditional methods for negation generation, i.e., pattern-based textual extractions and knowledge-graph-based inferences, as well as crowdsourced gold statements. We measure the correctness and salience of the generated lists about subjects from different domains. Our evaluation shows that guided probes do in fact improve the quality of generated negatives, compared to the zero-shot variant. Nevertheless, using both prompts, LLMs still struggle with the notion of factuality of negatives, frequently generating many ambiguous statements, or statements with negative keywords but a positive meaning.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Completeness, Recall, and Negation in Open-world Knowledge Bases: A SurveyACM Computing Surveys, 2024
- Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge2023
- UnCommonSense: Informative Negative Knowledge about Everyday Concepts2022
- Refined Commonsense Knowledge from Large-Scale Web ContentsIEEE Transactions on Knowledge and Data Engineering, 2022
- NegatER: Unsupervised Discovery of Negatives in Commonsense Knowledge Bases2021
- Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly2020
- Mining an "anti-knowledge base" from Wikipedia updates with applications to fact checking and beyondProceedings of the VLDB Endowment, 2019
- Commonsense Properties from Query Logs and Question Answering Forums2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019