OPT-13B (AI model): news timeline & CVEs

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

TACO reduces OPT-13B optimizer state 174× versus AdamW8bit and enables 30B fine-tuning on one H100.

TACO is a ternary column-wise one-sparse optimizer that follows Muon's operator-norm steepest-descent view while remaining compatible with AdamW-pretrained models. On OPT-13B it cuts persistent optimizer state from 27.7 GB to 0.16 GB (174× versus AdamW8bit) and peak training memory from 80.6 GB to 27.5 GB, with comparable accuracy and runtime. The method enables full-parameter fine-tuning of 30–32B models on a single 80 GB H100 across several families and tasks.

arXiv cs.AI / cs.LG / cs.CLOct 1, 17:59 UTC · 7d agoAI research

Probe-Space Preconditioning for Fast and Stable Zero-Order Training

1.5-SPSA preconditions zero-order probes so OPT-13B beats MeZO on SST-2 in 70 steps instead of 100,000.

The paper targets the convergence gap of zero-order optimization, which can train OPT-30B in inference mode at roughly 60GB instead of about 600GB for Adam with backpropagation. Reallocating compute toward larger probe batches lets 1SPSA beat MeZO with less training compute, and 1.5-SPSA adds one clean forward pass per step to form a cheap diagonal preconditioner. Across six post-training datasets on Qwen3 and OPT, 1.5-SPSA leads prior zero-order solvers; on OPT-13B SST-2 it gains 3.1 accuracy points over both MeZO and backpropagation in 70 steps versus MeZO's 100,000, and the implementation trains OPT-30B in place on A100 GPUs.

arXiv cs.AI / cs.LG / cs.CLSep 29, 17:43 UTC · 9d agoAI research

OPT-13B

Timeline2 articles

Appears with

TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning

Probe-Space Preconditioning for Fast and Stable Zero-Order Training