AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph
Authors:
- Haochen ShiHSHaochen Shi
- Weiqi WangWWWeiqi Wang
- Hongming ZhangHZHongming Zhang
Preprint
Abstract
Cognitive research indicates that abstraction ability is essential in human intelligence, which remains under-explored in language models. In this paper, we present AbsPyramid, a unified entailment graph of 221K textual descriptions of abstraction knowledge. While existing resources only touch nouns or verbs within simplified events or specific domains, AbsPyramid collects abstract knowledge for three components of diverse events to comprehensively evaluate the abstraction ability of language models in the open domain. Experimental results demonstrate that current LLMs face challenges comprehending abstraction knowledge in zero-shot and few-shot settings. By training on our rich abstraction knowledge, we find LLMs can acquire basic abstraction abilities and generalize to unseen events. In the meantime, we empirically show that our benchmark is comprehensive to enhance LLMs across two previous abstraction tasks.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- ASER: Towards large-scale commonsense knowledge acquisition via higher-order selectional preference over eventualitiesArtificial Intelligence, 2022
- Explaining Answers with Entailment Trees2021
- Adversarial NLI: A New Benchmark for Natural Language Understanding2020
- Learning Typed Entailment Graphs with Global Soft ConstraintsTransactions of the Association for Computational Linguistics, 2018
- Defining textual entailmentJournal of the Association for Information Science and Technology, 2018
- Abstraction in Different Disciplines2013
- The PASCAL Recognising Textual Entailment Challenge2006
- K-Lines: A theory of memoryCognitive Science, 1980