HotPOT pairs PDE-specific numerical experts in one operator transformer
Original title: HotPOT: Operator Transformer with Heterogeneous Physics Experts for Large-Scale PDE Pre-training
Across 24 matched comparisons, HotPOT cuts rollout error 50.7% on average versus MoE-POT, including a 17M model.
HotPOT is an operator transformer for large-scale PDE pre-training that uses a shared encoder and a neural-numerical expert per PDE family. Each expert pairs a learnable predictor with a fixed numerical update that has no trainable parameters and computes the next state. Across five PDE families and six settings, HotPOT beats MoE-POT in all 24 matched-scale comparisons before and after fine-tuning, reducing rollout error by 50.7% on average. A 17M model also surpasses substantially larger MoE-POT models, and gains extend to zero-shot prediction under unseen physical conditions.
- Experts use homogeneous-architecture routing only in prior MoE models; HotPOT uses heterogeneous numerical updates.
- The numerical update contains no trainable parameters.
- Gains hold before and after fine-tuning in every matched comparison.
- Code is at github.com/cruiseresearchgroup/HotPOT-code.
From the original article
1 min read at arxiv.orgLarge-scale PDE pre-training trains a single model on the data from multiple partial differential equations (PDEs). However, adding heterogeneous PDEs with different governing equations, physical variables, and variable interactions does not always improve predictive performance across tasks.
By Jiayi Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.