TRAVELER: A Benchmark for Evaluating Temporal Reasoning Across Vague, Implicit and Explicit References
Open Access
- 22 April 2026
- journal article
- Published by Springer Nature in SN Computer Science
- Vol. 7 (5)
Abstract
Temporal references to future, past or ongoing events are ubiquitous in natural language. Yet, there is no comprehensive benchmark dataset allowing to assess the ability of Large Language Models (LLMs) to correctly resolve these references. In this work we close this gap and provide such a benchmark that supports the evaluation of the interpretation of Explicit (e.g., “on 30.12.2024”), Implicit (e.g., “yesterday”) and Vague (e.g., “recently”) temporal references. We introduce TRAVELER, a synthetic question-answering benchmark that evaluates a system’s ability to perform temporal reasoning with respect to a set of past events. The benchmark comprises 3,300 English questions, categorized by their temporal reference and automatically generated via four templates from events sets that contain 5-100 events from a household domain. For the Vague category, ground-truth answers were established via human surveys on Prolific, following a procedure inspired by Kenneweg et al. As reported by Kenneweg et al. (in: Proceedings of the Workshop on Cognitive Aspects of the Lexicon @, Torino, 2024). LREC-COLING 2024). To demonstrate the benchmark’s applicability, we evaluate four state-of-the-art LLMs on it. All benchmarked LLMs can answer questions over events sets with a handful of events and Explicit temporal references successfully, but performance clearly deteriorates with larger event set length and when temporal references get less explicit. Notably, the Vague question category exhibits the lowest performance across all models. TRAVELER exposes limitations in current LLMs’ event temporal reasoning capabilities: Their performance clearly deteriorates with longer event sets and when including Vague temporal references. The benchmark, which is publicly available ( https://gitlab.ub.uni-bielefeld.de/s.kenneweg/TRAVELER ) also offers the possibility to test the event-temporal reasoning capabilities of other models beyond those tested in this work.Keywords
All Related Versions
Funding Information
- Universität Bielefeld
This publication has 34 references indexed in Scilit:
- Robot Tasks with Fuzzy Time Requirements from Natural Language Instructions2024
- CoPAL: Corrective Planning of Robot Actions with Large Language Models2024
- Faithful Temporal Question Answering over Heterogeneous Sources2024
- TIQ: A Benchmark for Temporal Question Answering with Implicit Time Constraints2024
- Complex Temporal Question Answering on Knowledge Graphs2021
- TempQuestions2018
- On the value of temporal information in information retrievalACM SIGIR Forum, 2007
- Vagueness and grammar: the semantics of relative and absolute gradable adjectivesLinguistics and Philosophy, 2007
- Michiel van Lambalgen and Fritz Hamm. The proper treatment of events. Explorations in Semantics, no. 4. Blackwell Publishing, Oxford, 2005, xii + 251 pp.The Bulletin of Symbolic Logic, 2006
- The Logical Form of Action Sentences2001