Pretraining data scale reverses Muon-AdamW fine-tuning gap
Beyond large pretraining budgets AdamW outperforms Muon after fine-tuning with the gap widening as model size grows.
Stories per day · 30 days
Relatedbenchmarks1data-quality1evaluation1instruction-tuning1large-language-models1llm-as-judge1optimizers1pretraining1temporal-reasoning1
6 stories, newest first · one card per story
Stories tagged #finetuning
Response ratio predicts instruction-tuning utility better than common proxies
Under a fixed finetuning budget, answer-dense examples beat common quality proxies on perplexity and LLM-as-judge scores.
LLM temporal benchmarks reward memorized time knowledge, study finds
Same-dataset finetuning gained 6.8 to 23.1 points, while same-mode transfer ranged from −3.2 to +1.1 points.
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Spurious forgetting hides old facts through a reversible shared shift before slower fact-specific erosion.
EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
EdgeGen synthesizes database-grounded edge-case tasks that improve tool-calling agents without human annotation.
Pick Your Poison: Learning to Select Poison Sets for Stronger LLM Backdoor Attacks
Poison set selection swings LLM backdoor attack success from 3% to 80%; SAILS boosts held-out success by 30 points.