ASAP evaluates agent behavior in base models before post-training
Original title: Evaluating Agentic Capabilities Before Post-Training: An End-to-End Approach
The framework keeps the completion interface and scores software-engineering and deep-research tasks across checkpoints.
The paper introduces ASAP, a lightweight way to elicit agentic behavior from pretrained base models without converting them into chat-style agents. It adapts existing scaffolds with structured trajectory continuation, short reference prefixes, and minimal recovery. Authors say it separates base models and pretraining checkpoints on software-engineering and deep-research benchmarks and exposes trajectories for exploration, editing, testing, recovery, and completion.
Unconfirmed: Discriminative performance is claimed by the authors; benchmark names and scores are not given.
- ASAP preserves the completion interface rather than building chat agents.
- Scaffolds use trajectory continuation, short prefixes, and minimal recovery.
- Trajectories cover exploration, editing, testing, recovery, and task completion.
- Authors compare pretraining-stage behavior with patterns after post-training.
From the original article
1 min read at huggingface.co
Agentic capabilities are increasingly important for modern LLMs, yet remain difficult to evaluate during pretraining. Existing approaches typically rely on static trajectory-derived tasks or likelihood-based proxies, which are lightweight but provide limited visibility into how base models behave in interactive environments.
By Jiarui Qin, Junjie Huang, Yunjia Xi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.