Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference
Attack breaks permutation-based model confidentiality in hybrid FHE inference, recovering all ResNet-20 linear layers exactly with d+1 queries per layer.
The paper shows output-permutation plus noise fails to protect model confidentiality in hybrid FHE inference: d+1 admissible queries recover an exact permutation-invariant summary of a d-input linear layer, and shuffle-model DP amplification premises cannot hold under correctness-bounded noise. The authors recovered all linear layers of a Safhire-style ResNet-20 end-to-end from TFHE transcripts with zero error, using 5,712 total queries. Exact per-layer recovery was also confirmed on pretrained ImageNet-scale CNNs and ViT-B/16. Leaked layer spectra enable model fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys inference utility.
- d+1 admissible queries recover exact permutation-invariant layer summary.
- All ResNet-20 linear layers recovered end-to-end with zero error.
- Confirmed on pretrained ImageNet-scale CNNs and ViT-B/16.
- Leaked spectra enable fingerprinting, lineage attribution, improved logit-based extraction.
- Shuffle-DP amplification premise fails under correctness-bounded noise.
From the original article
1 min read at arxiv.orgHybrid fully homomorphic encryption~(FHE) inference improves the practicality of private inference by letting the server evaluate linear layers homomorphically while the client decrypts and applies nonlinearities. Recent schemes attempt to protect model confidentiality by returning noisy, output-permuted responses and appealing to shuffle-model differential privacy~(DP).
By Jiseung Kim · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.