Stochastic DPO graph pruning speeds training about 1.4x
Original title: Stochastic Pruning of the DPO Computation Graph: 1.4x Faster Training and Lower Peak Memory
Branch pruning cut training time about 30% versus stock DPO, with slightly lower mean accuracy.
A paper introduces stochastic pruning of the direct preference optimization computation graph, independently sampling response positions, prefixes, and preference branches before those operations run. Inverse-probability weighting yields a conditionally unbiased single-branch gradient for the sampled objective. Controlled tests find position and sequence pruning mainly cut peak memory, while branch pruning is the main speedup, with benchmark-dependent changes in open-ended scores.
- Branch pruning gave a 1.426x step-time speedup over a concatenated DPO baseline.
- Training time fell about 30% versus stock DPO across two model families.
- Most output-memory savings came from an implementation that keeps full response supervision.
- Mean accuracy was slightly lower, and open-ended scores changed by benchmark.
From the original article
1 min read at arxiv.orgDirect preference optimization (DPO) incurs training costs in output scoring, sequence execution, and backpropagation, so reducing supervision alone does not necessarily save computation. We introduce a unified framework for stochastic pruning of the DPO computation graph that independently samples response positions for scoring, response prefixes for execution, and preference branches for backpropagation.
By Yubing Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.