SkyPilot says rack-local scheduling yields 27% more usable GPUs
Topology-aware placement, in-place recovery, and defragmentation keep multi-node jobs on NVL72 NVLink instead of InfiniBand.
Stories per day · 30 days
1 RSS
Relatedfedavg1federated-learning1fsdp1gpu-scheduling1model-selection1nccl1nvl721nvlink1nvswitch1rdma1skypilot1
14 stories, newest first · one card per story
Stories tagged #distributed-training
CovFedAvg selects federated models using local low-loss curves
On width-16 gas-sensor clients, balanced accuracy rose from FedAvg's 88.88% to 89.84%, below matched fine-tuning.
TIDE speeds FSDP2 with zero-copy layout-aware collectives
On 64 GPUs, step time falls by up to 27.1% versus stock FSDP2 and 6.0% versus FSDP2+SymmMem on three training workloads.
DisaggKD splits teacher and student across nodes for distillation
On the Qwen3.5-35B-A3B to 9B pair, the split matches colocated LM loss and is faster.
FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention
FlashMask-3 adds Hopper and distributed mask-aware attention, raising long-context training throughput over prior kernels.
NeMo-DCR: Bit-Exact Delta-Compressed Refit for Scalable Agentic RL at Trillion-Parameter Scale
NeMo-DCR bit-exactly syncs sparse weight deltas, cutting trillion-parameter refits from 87.5 minutes to 150 seconds.
Accelerating Sharded Data Parallelism at Scale with Federated Learning
Hybrid FL+FSDP and FL+HSDP algorithms cut communication overhead in sharded data parallelism, accelerating Llama3.1 8B pre-training on 512 A100 GPUs by up to 8x.
Bridging the Gap Between Homogeneous and Heterogeneous Asynchronous Optimization Is Surprisingly Difficult
Lower bounds show heterogeneous asynchronous optimization cannot match homogeneous rates under standard similarity assumptions; strong interpolation plus local PL condition closes the gap.
Privacy-enhanced federated learning via asynchronous aggregation and local differential perturbation
Federated learning framework combining dynamic differential privacy, homomorphic encryption, and local DP retains 82.6% accuracy at epsilon 0.1 while cutting communication 21.3%.
Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
Four scheduling techniques bound all memory peaks in long-context MoE training, enabling 120B-667B models at 1M-token context with up to 10.4x throughput.
Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL
Hugging Face blog describes running async GRPO reinforcement learning with LoRA across HF Jobs using a storage bucket and proxy instead of NCCL.
HybridFLow: SDN-Orchestrated Client Partitioning for Hybrid Federated Learning
HybridFLow uses SDN topology visibility to partition federated-learning clients into sync/async groups, reaching 80% accuracy 33-40% faster than SmartFLow.
Miles v0.1: Production-Level Post-Training
Radix Ark open-sources Miles v0.1, a full-stack RL post-training framework demonstrated with asynchronous agentic RL on GLM-5.2 744B-A40B across 64 GB300 GPUs.
Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
An empirical study shows distributed Kolmogorov-Arnold Network training reaches 74.7% parallel efficiency at 8 A100 GPUs, with overheads driven by All-Reduce choices.