AgentTime: Can Agents Estimate and Control Their Own Runtime?
AgentTime finds coding agents often miss requested runtimes; Fable 5.1 deviates 2.9× versus Astra's 1.2×.
AgentTime is a benchmark of 222 tasks from 18 sources spanning coding, computer use, agentic work, and automated research. It tests whether agents work for a requested duration, forecast runtime, and estimate elapsed time afterward. Fable 5.1 in Claude Code typically misses requested runtimes by 2.9×, compared with 1.2× for GPT-6 Astra in Codex. Of 158 reviewed Astra runs, 14 explicitly slept after appearing to finish, and removing temporal information more than doubled retrospective error for Sol and Astra and nearly doubled it for Fable.
- Benchmark covers 222 tasks drawn from 18 sources.
- Fable 5.1 in Claude Code typically deviates by 2.9×.
- GPT-6 Astra in Codex deviates by about 1.2×.
- Fourteen of 158 Astra runs slept after seeming finished.
- Removing time cues roughly doubles retrospective estimation error.
Full article237 words · extracted from arxiv.org · click to collapse
An essential control of AI agents is their ability to manage runtime. This ability requires a sense of time-awareness, to predict and estimate wall-clock time and to control their own actions. Prior work has focused on time-awareness, but duration-following and control in native agent harnesses remain unexplored. We present AgentTime, a benchmark for testing whether agents can work for a requested duration, predict their runtime, and estimate elapsed time afterward. It comprises 222 tasks from 18 sources spanning coding, computer use, agentic work, and automated research. Duration-following experiments append a single instruction specifying how long to work, with requests ranging from about a minute to multiple days. Accuracy on these instructions varies substantially: Fable 5.1 in Claude Code deviates from requested runtimes by a typical factor of 2.9$\times$, compared with only 1.2$\times$ for GPT-6 Astra in Codex. However, matching the requested runtime does not, by itself, establish continued work on the task. Among 158 reviewed Astra runs with classifiable transcripts, 14 explicitly slept after appearing to finish. In forecasting experiments, predictions tend to overestimate natural runtimes. In retrospective experiments, removing temporal information more than doubles deviation for Sol and Astra and nearly doubles it for Fable. An agent's ability to complete a task does not guarantee that it can control its own time or work for the whole requested duration. For agents to run reliably, safely, and autonomously over long horizons, we require the evaluation of both.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2610.09944