Skip to content
NewA calmer, faster ZeroHour

SlideDP: Scaling Host-Resident LLM Fine-Tuning Across Multiple GPUs

Hugging Face daily papersby Ruijia Yang, Shiyuan Lin, Yulong Ao ()1 min read5 · 1 read
AI summary · grok-4.7Summaries by Vani AI

SlideDP scales host-resident multi-GPU LLM fine-tuning, beating offload baselines and nearing FSDP2 on Qwen3-14B.

SlideDP is a synchronous data-parallel runtime that keeps one authoritative host state and pipelines parameter delivery, gradient aggregation, and CPU updates for full-parameter LLM fine-tuning beyond GPU memory. Matched-batch sweeps report geometric-mean throughput 1.46-2.64 times SlideFormer, MegaTrain, and ZeRO-Offload. On four H100s it approaches GPU-resident FSDP2 for Qwen3-14B and, with a larger batch, processes over 1 million tokens per step, 11.2% above FSDP2's measured peak. It also supports 256K-token sequences for that model and fine-tunes Qwen2.5-72B on four RTX 4090s.

Key facts
  • Keeps one host state and pipelines parameters, gradients, and CPU updates
  • Geometric-mean throughput is 1.46-2.64x over three offload baselines
  • On four H100s, exceeds FSDP2 peak throughput by 11.2%
  • Fine-tunes Qwen2.5-72B on four RTX 4090 GPUs
Explain this AI news with Vani AIWhat was announced, what is new and why it matters, in a new chat

From the original article

1 min read at huggingface.co

Host-resident layer streaming enables full-parameter LLM fine-tuning beyond GPU memory, but data-parallel ranks compete for shared host resources. Replicated transfers amplify traffic, while strong scaling can expose host work as computation windows shrink. We present SlideDP, a synchronous data-parallel runtime for shared-host multi-GPU systems.

Hugging Face daily papers

By Ruijia Yang, Shiyuan Lin, Yulong Ao · huggingface.co

Read the full article at Hugging Face daily papers

A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.