Accelerating Sharded Data Parallelism at Scale with Federated Learning
Hybrid FL+FSDP and FL+HSDP algorithms cut communication overhead in sharded data parallelism, accelerating Llama3.1 8B pre-training on 512 A100 GPUs by up to 8x.
The paper introduces FL+FSDP and FL+HSDP, hybrid algorithms that interleave sharded data parallelism with FedAvg-style federated aggregations to decouple large DP deployments into loosely-coupled federation groups. Formal communication-cost analysis and experiments on multi-tier GPU interconnects demonstrate scalability and flexibility. A Llama3.1 8B pre-training run on 512 A100 GPUs achieves up to 8.04x faster data processing and 4.48 lower evaluation perplexity than sharded DP baselines under identical hyperparameters.
30