Scalability Analysis of Distributed Kolmogorov-Arnold Network Training on High-Performance Computing Systems
An empirical study shows distributed Kolmogorov-Arnold Network training reaches 74.7% parallel efficiency at 8 A100 GPUs, with overheads driven by All-Reduce choices.
The study evaluates data-parallel Kolmogorov-Arnold Network (KAN) training on the FinisTerrae III supercomputer using up to 8 NVIDIA A100 GPUs across 4 nodes with PyTorch Distributed Data Parallel. Strong scaling yields 5.97x speedup and 74.7% parallel efficiency at 8 GPUs, comparable to conventional deep learning workloads, while communication overhead ranges from 1.3% to 6.1%, driven mainly by All-Reduce algorithm selection and inter-node latency rather than KAN's edge-wise gradient structure. Weak scaling shows an initial single-to-multi-GPU throughput drop followed by stability, and the parameter-to-memory ratio improves with model size even as training time scales unfavorably. The authors provide GPU topology and model-size deployment guidelines for KAN training.
- KANs replace MLP activations and linear weights with learnable univariate edge functions
- 74.7% parallel efficiency and 5.97x speedup achieved on 8 A100 GPUs across 4 nodes
- Communication overhead of 1.3-6.1% stems from All-Reduce selection, not KAN gradients
- Parameter-to-memory ratio improves with model size while training time scales poorly
- Deployment guidelines cover GPU topology and model-size selection
- Vendors
- NVIDIA
- Organizations
- FinisTerrae III operators
- AI models
- Kolmogorov-Arnold Network
From the original article
1 min read at arxiv.orgKolmogorov-Arnold Networks (KANs) replace the fixed activation functions and linear weights of Multi-Layer Perceptrons (MLPs) with learnable univariate functions on network edges, offering improved interpretability and, in some settings, competitive parameter efficiency.
By Guangneng Chen · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.