TIDE speeds FSDP2 with zero-copy layout-aware collectives
Original title: The Endgame of FSDP Communication: Layout-Aware Collectives on Copy Engines, RDMA, and NVSwitch
On 64 GPUs, step time falls by up to 27.1% versus stock FSDP2 and 6.0% versus FSDP2+SymmMem on three training workloads.
The paper says FSDP2 layout conversion between per-parameter tensors and flat buffers costs 6.4% of step time for Llama-3.1 70B at 8K tokens per GPU, and a default NCCL reduce-scatter can slow a concurrent GEMM by 69%. TIDE is a set of zero-copy collectives on public NCCL 2.31+ APIs that keep FSDP2 storage while placing data through NVSwitch, RDMA, and copy engines. TIDE-RS also reduces per-parameter bf16 gradients into fp32 shards, a mixed-precision mode NCCL does not provide. Measured workloads are Llama-3.1 70B, Wan2.2, and Qwen3-30B-A3B.
- TIDE-AG writes shards straight into parameter storage without SMs for transport.
- Reduce-scatter SM time drops by factors of 12.5 to 14.5.
- The reduce-scatter buffer stays nearly constant from 16 to 64 GPUs.
- Peak memory is up to 4.7 GiB below FSDP2+SymmMem.
- Vendors
- NVIDIA
- AI models
- Llama-3.1 70BWan2.2Qwen3-30B-A3B
From the original article
1 min read at arxiv.orgFully Sharded Data Parallelism (FSDP2) maintains parameters and gradients as per-parameter tensors, whereas its collective path operates on flat, rank-major buffers. Converting between these layouts copies all-gather outputs into parameter storage and packs gradients into full-group fp32 buffers. Layout conversion alone accounts for 6.
By Zejian Wang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.