An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference
Odin runs Llama-3-8B fully homomorphic encrypted inference on a single H100 in 366 seconds, a 4.51x speedup over THOR.
Odin is an open-source end-to-end GPU CKKS implementation for privacy-preserving Llama-3-8B inference that co-designs ciphertext packing with model execution. A feature-major cross-layer layout unifies residual connections and layer interfaces, while transient intra-operator layouts serve linear projections and attention, avoiding intermediate repacking of QK^T softmax outputs. Minimax polynomial approximation with input-range control reduces polynomial degree and multiplicative depth for nonlinear ops. With 128-token input, Odin evaluates all 32 Transformer layers on one NVIDIA H100 80 GB in 366.4 s using 58.9 GiB peak memory, versus 1651.9 s for the THOR baseline, a 4.51x speedup.
- First open-source end-to-end GPU CKKS FHE implementation of Llama-3
- Evaluates all 32 layers of Llama-3-8B on a single NVIDIA H100 80 GB
- 366.4 s end-to-end and 58.9 GiB peak memory for a 128-token input
- 4.51x speedup over THOR baseline under identical model, parameters, and hardware
From the original article
1 min read at arxiv.orgCloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots.
By Yuhang Fan · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.