MOAI speeds non-interactive homomorphic transformer inference
Original title: MOAI: Module-Optimizing Architecture for non-interactive secure transformer Inference
It cuts BERT-base evaluation time 52.8% versus THOR and reports 2.36 minutes amortized per input on one GPU.
Researchers propose MOAI, a homomorphic-encryption framework for non-interactive secure transformer inference on encrypted inputs. Column and diagonal packing, rotation-free Softmax and LayerNorm, and interleaved batching cut costly HE rotations. Versus THOR, BERT-base evaluation time falls 52.8%; on Powerformer the reduction is 55.7%. The authors also apply MOAI to LLaMA-3-8B and release the implementation as open source.
- Rotation-free Softmax and LayerNorm remove 2,448 HE rotations in BERT-base inference.
- MOAI uses at least 1.7x fewer HE rotations than prior work on BERT-base matrix multiplications.
- Amortized time is 2.36 minutes per input on a single GPU.
- Evaluation-time cuts are 52.8% versus THOR and 55.7% versus Powerformer.
From the original article
1 min read at arxiv.orgPrivacy concerns have been raised in Large Language Models (LLM) inference when models are deployed in Cloud Service Providers (CSP). Homomorphic encryption (HE) offers a promising solution by enabling secure inference directly over encrypted inputs. However, the high computational overhead of HE remains a major bottleneck.
By Linru Zhang · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.