Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
ZFO fine-tunes language models by pairing a first-order direction with cheap zeroth-order step search.
Zero-and-First-Order optimization (ZFO) uses a trusted first-order optimizer for the update direction and two extra objective evaluations to choose a curvature-aware step along that line. Shared-sample finite differences estimate curvature, and theory shows a near-optimal step and convergence near a stationary point. Across language models and datasets, ZFO often improves optimization and final performance versus fixed-step baselines. Code is public.
- A first-order optimizer sets direction; step size is searched in one dimension.
- Two extra objective evaluations build a local curvature model.
- Theory covers reliable curvature estimates and convergence near stationarity.
- ZFO often beats fixed-step first-order baselines on language models.
From the original article
1 min read at arxiv.orgStep-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size.
By Cristian McGee · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.