LLM temporal benchmarks reward memorized time knowledge, study finds
Original title: Keeping Time vs. Understanding Time: Neither Alone Explains Temporal Reasoning in LLMs
Same-dataset finetuning gained 6.8 to 23.1 points, while same-mode transfer ranged from −3.2 to +1.1 points.
A paper tests the common split of LLM temporal reasoning into arithmetic and semantic modes. Finetuning on benchmark subsets labeled as one mode mostly helped only that dataset, with little transfer to another dataset in the same mode. Cognitive-demand evaluation found knowledge, not quantitative or logical reasoning, was the main driver of benchmark scores. The authors conclude current benchmarks do not cleanly realize the arithmetic-semantics distinction.
- Transfer within a purported mode ranged from −3.2 to +1.1 percentage points.
- Dataset-specific finetuning gains were +6.8 to +23.1 percentage points.
- Quantitative and logical reasoning were not the primary performance drivers.
- Two benchmark families were used in the transfer analysis.
From the original article
1 min read at arxiv.orgTemporal reasoning is widely assumed to decompose into two modes: temporal arithmetic (date and time calculations), and temporal semantics (understanding temporal and causal relations). This dichotomy shapes how benchmarks are designed, how models are trained, and how progress is measured, yet its empirical validity has never been directly tested. We provide two lines of evidence that challenge it.
By Auss Abbood · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.