Latent-MoE: Domain-Aware Mixture-of-Experts for PDEs with Multi-Regime Physics
Latent-MoE adds domain-aware mixture-of-experts routing to PINNs, improving accuracy over an order of magnitude on multi-regime physics PDEs.
The paper shows standard coordinate networks used in physics-informed neural networks have translation-variant NTKs causing long-range gradient conflicts on PDEs with spatially varying physics. MoE architectures with centered compact-support routers produce a uniformly banded NTK that localizes learning. Latent-MoE interleaves domain-aware MoE blocks in a shared backbone, outperforming FB-PINNs and X-PINNs by over an order of magnitude on multi-stage time-variable physics benchmarks.
- NTK analysis explains PINN failure on multi-regime PDEs
- Compact-support routers yield exponentially decaying kernel-regression weights
- Latent-MoE allows capacity sharing across regions via shared backbone
- Improves over an order of magnitude versus global and rigid-decomposition baselines
Full article186 words · extracted from arxiv.org · click to collapse
Physics-informed neural networks (PINNs) struggle on PDEs whose governing physics varies across the domain. We trace this to a structural property of standard coordinate networks: their neural tangent kernel (NTK) is translation-variant and lets training points of large coordinate magnitude disproportionately influence predictions elsewhere, producing long-range coupling and gradient conflict during training. We show analytically and empirically that mixture-of-experts (MoE) architectures with centered, compact-support routers yield a uniformly banded NTK whose kernel-regression weights decay exponentially with distance, localizing the learning. Building on this, we propose \emph{Latent-MoE}, which interleaves domain-aware MoE blocks within a shared backbone. Unlike FB-PINNs or X-PINNs, which rigidly partition both the domain and the parameters so that the parameters on different subdomains are updated independently, Latent-MoE is designed to preserve the localization benefit of domain-aware routing while allowing capacity to flow across regions through the shared backbone. On standard homogeneous-physics benchmarks Latent-MoE is competitive with established baselines; on benchmarks with multi-stage time-variable physics, where global models and rigid domain decompositions both fall into spurious solutions, it improves over them by more than an order of magnitude, with markedly reduced gradient conflict during training.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.07814