Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models
QPriv-VL prunes privacy-sensitive visual tokens in federated/split VQA, cutting membership-inference success on VQA-RAD from 0.99 to 0.76-0.79 using ~40% of tokens.
The paper proposes QPriv-VL, a question-guided token-pruning framework for federated, split, and U-shaped split learning that suppresses privacy-sensitive visual patches before transmission. Its Dynamic Threshold Predictor combines cross-modal question relevance with frozen DINOv2-derived sensitivity to compute a per-sample pruning ratio and retention mask in one forward pass, without sensitivity labels. Evaluated on GQA, OK-VQA, VQAv2, SLAKE, VQA-RAD, and PathVQA against FSHA, FORA, iDLG, and attribute-inference membership inference attacks, it matches or beats fixed-ratio pruning. On VQA-RAD it reduces membership-inference success from 0.99 to 0.76-0.79 while preserving competitive accuracy with about 40% of the original token budget.
- Dynamic Threshold Predictor outputs per-sample pruning ratio and token mask in one pass
- Combines question relevance with frozen DINOv2 sensitivity, requiring no sensitivity labels
- Cuts VQA-RAD membership-inference success from 0.99 to 0.76-0.79
- Preserves competitive VQA accuracy using about 40% of visual-token budget
From the original article
1 min read at arxiv.orgVisual Question Answering (VQA) with Vision-Language Models (VLMs) is increasingly used in privacy-sensitive and bandwidth-constrained settings. Federated Learning (FL), Split Learning (SL), and U-Shaped Split Learning (USL) keep raw data local, but transmitting all visual tokens across a model partition remains costly and can expose private information.
By Md Khalid Syfullah · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.