Paper proves orthogonal symmetry of the differential neural tangent kernel
Original title: The Symmetry of Differential Neural Tangent Kernel under the Orthogonal Group
Operators from the same group orbit share spectra, and shallow RePU networks get an eigenvalue decay rate.
The paper studies symmetry of the infinite differential neural tangent kernel under an orthogonal group for two-layer and deep networks. Results cover activations with or without bias and use the kernel's explicit expression or a recursive formula. Integral operators from the same group orbit share a spectrum, and their reproducing kernel Hilbert spaces differ only by an orthogonal transformation. For symmetric operators, the fixed kernel is zonally symmetric on the unit sphere.
- Eigenvalue decay is derived for shallow RePU networks with zero-bias initialization.
- The analysis applies to both shallow and deep overparameterized PINNs.
- Results hold for activations with and without a bias term.
From the original article
1 min read at arxiv.orgThe differential neural tangent kernel (DNTK) is a tool for analyzing the training dynamics of overparameterized physics informed neural networks. In this work, we investigate the symmetry of the infinite DNTK under an orthogonal group, for two layer and deep neural networks separately, based on the explicit expression of the DNTK and recursive formula, respectively.
By Bangti Jin · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.