Bonbon shows zero-shot molecular skills without structural training
A 1.6B model trained only on protein-ligand sequences reports binding-site, mechanism, and bond predictions plus a 61.5% wet-lab hit rate.
Stories per day · 30 days
2 RSS
Relateddiffusion2pretraining2representation-learning2robotics2transformers2vlm2activation-monitoring1agents1auroc1autoregressive1bptt1continued-pretraining1dinov21embodied-agents1embodied-ai1ensembling1fid1fine-tuning1
31 stories, newest first · one card per story
Stories tagged #scaling-laws
Multi-Epoch Pretraining Scaling Theory Shows Ensembling Benefits
Logit averaging in ensembles reduces late-training overfitting and delays optimal stopping in fixed-data pretraining.
Hierarchical Residuals Decouple Residual and Compute Width
Block-wise grouping with wider residuals and learned projections improves FLOPs efficiency while preserving standard layer structure.
Paper turns the Platonic Representation Hypothesis into a JEPA theorem
A 109M-parameter text JEPA beat language models up to 72B on alignment with DINOv2 and CLIP image encoders.
SynthAPI scaling law links tool-use gains to environment diversity
Across 458,752 tasks and 849 models, tool and database diversity beat adding tasks as data budgets grow.
Paper splits visual generation into autoregressive knowledge and diffusion rendering
A 3B text-to-image model trained in 6,600 H100 hours scored 86 on GenEval and 87 on DPG-Bench.
Safety probe error follows scaling laws in size and labels
Laws fitted on a family's smallest models predict the next model's error within 0.02 in 72% of cases.
Scaling law favors full BPTT over truncated looped transformers
Two fully trained loops beat the compute-optimal non-looped model with about 27% fewer non-embedding parameters.
Continued pre-training study scales embodied agents across virtual worlds
One hundred twenty runs span 0.8B–27B models, 50–1,600 hours of data, and five world mixtures.
Paper studies physical recursive self-improvement in VLM policies
PhysRSI-eval scores VLMs on 18 manipulation tasks under a fixed budget and tests whether public scores predict later physical gains.
Paper studies physical recursive self-improvement in VLM agents
PhysRSI-eval scores 18 manipulation tasks and finds close public averages can hide late or inefficient gains.
Minimax-optimal Scaling Laws in Ridge Regression via Transfer Learning
Transfer from ridge-regression pretraining can achieve minimax-optimal scaling under power-law covariance shift.
Generative Representation Learning for Scalable Latent Tokenization
A generative tokenizer avoids representation collapse and reaches FID-50K 2.18 with JiT-L.
Why AlphaFold Didn't Solve Protein Folding — Pushmeet Kohli, Google DeepMind & Sal Candido, Biohub
DeepMind and Biohub researchers say AlphaFold did not solve protein folding and biology needs better data, not just scale.
Multi-Stream Attention Residuals
MSAR adds multi-head depth attention to residuals, with a 1.58× iso-loss speedup over standard residuals.
RoboJEPA: Scaling Robotic Latent World Models
RoboJEPA, an 8B JEPA world model on 12 embodiments, reports compute scaling laws for robotic planning.
How to scale your HEP ML models: A recipe for robust architecture comparisons at scale
Researchers give a recipe for robust scaling-law comparisons of HEP models on ATLAS JetSet2.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model looped recurrence and MoE sparsity, matching larger models at equal compute.
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
Study of 800 models finds wild AI web text initially helps data-starved pretraining, then raises human-text loss.
Scaling Laws for Looped Mixture of Experts
New scaling laws jointly model recurrence and sparsity, predicting looped MoE loss and guiding compute-efficient design.
How Much Is an AI Token Worth? Scaling Laws for Wild AI-Generated Web Text
A study of 800 models finds wild AI web text can hurt pretraining and proposes a new scaling law.
ScAn-Bench: Evaluating Scaling Analysis Methodology
ScAn-Bench evaluates how researchers fit and extrapolate scaling laws using thousands of LLM and VLM checkpoints.
How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretraining
Scaling laws suggest encoder-free multimodal models catch encoder-based ones near 10^22 FLOPs as the visual prior matters less.
Training and Inference Dynamics of PLDR-LLMs: Row-Map Collapse, Renormalization, and Predictive Reduction
A monograph derives exact training and inference identities for power-law decoder representation language models.