SPA approximates gated activations for faster FHE inference
Original title: SPA: Structure-Preserving Approximation of Gated Activations under FHE
On [-5,5], SPA-balanced is 3.25× faster than MOAI for erf-GELU at comparable error.
Nonlinear activations remain a bottleneck for neural inference under fully homomorphic encryption. SPA keeps the exact odd part of activations of the form x times g(x) and approximates only the even residual as a polynomial in x squared. Plaintext tests on Gemma-4-E2B and Llama-3.2-3B largely preserve task performance without weight updates, and CKKS benchmarks report better error or latency than MOAI, Euston, and THOR.
Unconfirmed: Speed and error figures are the paper's CKKS benchmark claims.
- Covered activations include GELU, SiLU, fixed-parameter Swish, and GEGLU/SwiGLU branches.
- SPA-accurate cuts QuickGELU RMSE 79.22% and is 10.40× faster than Euston.
- For SiLU, it reports 19.02% lower RMSE and a 2.06× speedup over THOR.
- Tests use Gemma-4-E2B and Llama-3.2-3B without weight updates.
From the original article
1 min read at arxiv.orgNonlinear activation evaluation remains a major bottleneck in neural inference under fully homomorphic encryption (FHE). FHE supports efficient addition and multiplication, but lacks native operations for exponential functions, reciprocals, and value-dependent selection. High-degree polynomial approximations on fixed intervals are therefore commonly used to replace nonlinear activations.
By Rui Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.