LLMs Apply Time-Dependent Constraints in Only 30.5% of Cases
Original title: Before You Continue: Evaluating Time-Sensitive Safety Awareness in Conversational Assistants
Thirteen models correctly calculate elapsed time in 62.1% of requests but fail to apply constraints.
Researchers created TASC benchmark with 3,418 inputs from 128 templates plus 33 scenarios to test whether LLMs recognize and adjust for time-dependent constraints in conversations. Across 13 models, assistants account for constraints in only 30.5% of cases where the action violates a rule while rarely warning users unnecessarily. Models fail to apply constraints in 62.1% of cases despite correctly calculating elapsed time. Explicitly providing timing resolves only about half of these failures.
- TASC benchmark has 3,418 inputs from 128 templates
- Separate set of 33 scenarios tests elapsed time calculation
- Models apply constraints in 30.5% of violating cases
- Explicit timing resolves only half of failures
- Gap between temporal reasoning and application
From the original article
1 min read at arxiv.orgLarge language model (LLM) assistants increasingly help users with everyday activities that involve time-dependent constraints. However, users may not mention these constraints in later requests, requiring assistants to recognize and adjust their advice accordingly.
By Peidi Dong · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.