CLadder: A Benchmark to Assess Causal Reasoning Capabilities of Language Models
Preprint
Abstract
The ability to perform causal reasoning is widely considered a core feature of intelligence. In this work, we investigate whether large language models (LLMs) can coherently reason about causality. Much of the existing work in natural language processing (NLP) focuses on evaluating commonsense causal reasoning in LLMs, thus failing to assess whether a model can perform causal inference in accordance with a set of well-defined formal rules. To address this, we propose a new NLP task, causal inference in natural language, inspired by the "causal inference engine" postulated by Judea Pearl et al. We compose a large dataset, CLadder, with 10K samples: based on a collection of causal graphs and queries (associational, interventional, and counterfactual), we obtain symbolic questions and ground-truth answers, through an oracle causal inference engine. These are then translated into natural language. We evaluate multiple LLMs on our dataset, and we introduce and evaluate a bespoke chain-of-thought prompting strategy, CausalCoT. We show that our task is highly challenging for LLMs, and we conduct an in-depth analysis to gain deeper insights into the causal reasoning abilities of LLMs. Our data is open-sourced at https://huggingface.co/datasets/causalNLP/cladder, and our code can be found at https://github.com/causalNLP/cladder.Keywords
All Related Versions
This publication has 33 references indexed in Scilit:
- Can Large Language Models Transform Computational Social Science?Computational Linguistics, 2024
- Large language models encode clinical knowledgeNature, 2023
- Evaluating vaccine allocation strategies using simulation-assisted causal modelingPatterns, 2023
- Causal inference and data fusion in econometricsThe Econometrics Journal, 2023
- How Can We Know What Language Models Know?Transactions of the Association for Computational Linguistics, 2020
- Largest Measles Epidemic in North America in a Decade—Quebec, Canada, 2011: Contribution of Susceptibility, Serendipity, and Superspreading EventsThe Journal of Infectious Diseases, 2012
- How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project StarThe Quarterly Journal of Economics, 2011
- Causality and Counterfactuals in the Situation CalculusJournal of Logic and Computation, 2007
- Causes and Explanations: A Structural-Model Approach. Part I: CausesThe British Journal for the Philosophy of Science, 2005
- Causal diagrams for empirical researchBiometrika, 1995