Shuffling is Not Enough: Breaking Permutation-Based Model Confidentiality in Hybrid FHE Inference
Attack breaks permutation-based model confidentiality in hybrid FHE inference, recovering all ResNet-20 linear layers exactly with d+1 queries per layer.
The paper shows output-permutation plus noise fails to protect model confidentiality in hybrid FHE inference: d+1 admissible queries recover an exact permutation-invariant summary of a d-input linear layer, and shuffle-model DP amplification premises cannot hold under correctness-bounded noise. The authors recovered all linear layers of a Safhire-style ResNet-20 end-to-end from TFHE transcripts with zero error, using 5,712 total queries. Exact per-layer recovery was also confirmed on pretrained ImageNet-scale CNNs and ViT-B/16. Leaked layer spectra enable model fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys inference utility.
- d+1 admissible queries recover exact permutation-invariant layer summary.
- All ResNet-20 linear layers recovered end-to-end with zero error.
- Confirmed on pretrained ImageNet-scale CNNs and ViT-B/16.
- Leaked spectra enable fingerprinting, lineage attribution, improved logit-based extraction.
- Shuffle-DP amplification premise fails under correctness-bounded noise.
Full article168 words · extracted from arxiv.org · click to collapse
Hybrid fully homomorphic encryption~(FHE) inference improves the practicality of private inference by letting the server evaluate linear layers homomorphically while the client decrypts and applies nonlinearities. Recent schemes attempt to protect model confidentiality by returning noisy, output-permuted responses and appealing to shuffle-model differential privacy~(DP). We show that this protection fails in the correctness regime required by hybrid FHE systems. For a $d$-input linear layer, $d+1$ admissible queries suffice for exact recovery of a permutation-invariant layer summary, hence for perfect model distinguishability. We further show that input DP is orthogonal to model confidentiality and that the local-DP premise required for shuffle amplification cannot hold under correctness-bounded noise. We recover all linear layers of a \safhire{}-style ResNet-20 end-to-end from TFHE transcripts with zero error, using $d+1$ queries per layer for a total of $5{,}712$ direct queries. Under the same query model, we also confirm exact per-layer recovery on pretrained ImageNet-scale CNNs and ViT-B/16. The leaked spectra enable fingerprinting, lineage attribution, and improved logit-based extraction, while suppressing them destroys inference utility.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.12911