FP4 pretraining needs more exposures to learn rare facts
NVFP4 needed 1.32× and 1.52× as many exposures as BF16 to match recall while loss differed by 0.5% and 2%.
Stories per day · 30 days
3 RSS
Relatedlanguage-models5mixture-of-experts5benchmarks4continual-learning3foundation-models3optimizers2reinforcement-learning2scaling-laws2supervised-fine-tuning2transformers2agent1agent-evaluation1attention1autoregressive1auto-research1bf161code1conditional-prediction1
56 stories, newest first · one card per story
Stories tagged #pretraining
Channels4World releases 100,000 outdoor RF scenes for wireless models
The 29.14-TB corpus spans 3.5 and 7 GHz with a 90K/10K scene-level train-test split.
Pretraining data scale reverses Muon-AdamW fine-tuning gap
Beyond large pretraining budgets AdamW outperforms Muon after fine-tuning with the gap widening as model size grows.
Multi-Epoch Pretraining Scaling Theory Shows Ensembling Benefits
Logit averaging in ensembles reduces late-training overfitting and delays optimal stopping in fixed-data pretraining.
DepthGraft shares KV hierarchically to regularize deep LLMs
On a 16B MoE model the authors report +7.0 points on MMLU-Pro while cutting KV cache by about one third.
DPDM Improves Agent Decision Making via Shared Predictor
DPDM raises mean accuracy from 78.80% to 84.92% across 50 benchmarks using one frozen predictor.
Hierarchical Residuals Decouple Residual and Compute Width
Block-wise grouping with wider residuals and learned projections improves FLOPs efficiency while preserving standard layer structure.
Paper unifies foundation-model pretraining, SFT and RL under one algorithm
A single likelihood trainer mixes static data and RL generations and is reported to improve the accuracy-compute tradeoff.
Paper unifies pretraining, SFT, and RL under one likelihood algorithm
The authors recast RL stages as maximum-likelihood fits to model-generated targets and mix them in one trainer.
AIR schedule targets effective learning rate, not nominal rate
On 128 billion-token MoE pretraining, AIR lowers terminal training loss by about 0.01 versus cosine at matched cumulative learning rate.
HotPOT pairs PDE-specific numerical experts in one operator transformer
Across 24 matched comparisons, HotPOT cuts rollout error 50.7% on average versus MoE-POT, including a 17M model.
MuonGS relaxes weight-norm control for LLM pretraining
On dense models it reports about 15% token-equivalent speedup versus MuonH at matched validation loss.
Point-in-time models tie temporal variability to later pretraining gains
High-TVS items beat low-TVS items in 34 of 36 settings by 7.3 chance-normalized accuracy points on average.
Auto-Research Arena benchmarks autonomous language-model research
A sequential search baseline beat several complex harnesses; reused ideas lowered loss under matched FLOPs.
Unigram cross-entropy tracks zero-shot scores from pre-training overlap
Across 10 benchmarks, four corpora, and three scales, lower UCE predicted higher accuracy, and document selection and prompt routing also gained.
Nestor nested sparsity beats MoE transformers at equal compute
Attention-side sparsity nests with mixture-of-experts sparsity and adapts to new domains with less forgetting.
Halcyon trains text-to-image models without language pretraining
Authors say Halcyon-1.6B beats a JiT baseline on gFD with 275 times less pretraining compute.
Hybrid models suppress attention retrieval unless recurrence updates pause
A staged pretraining procedure that freezes recurrence mid-training improves loss, reasoning, and retrieval on models up to 1B parameters.
ECHO folds multi-token prediction into the backbone itself
Dynamic Depth Sampling draws training depth with decaying probabilities, at half the overhead of training every depth.
Data-Efficient Classifier Pretraining Through Diffusion Models
DiffPT uses diffusion pretraining to improve low-data classifiers, especially on medical images and time series.
Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
NVIDIA describes scaling Indic pretraining to 2.8 trillion tokens with an open data pipeline.
VLA-Δ: Learning Physical State Transition for Vision-Language-Action Models
VLA-Δ trains robot policies on physical state transitions and beats π0.5 in real-robot tests.
Learning to Read Out: Unembedding Dynamics in Language Model Pretraining
Pythia encodes task information in hidden states before its unembedding readout can express it.
ASAP evaluates agent behavior in base models before post-training
The framework keeps the completion interface and scores software-engineering and deep-research tasks across checkpoints.