Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection
UCF-Net fuses CLIP and DINO features with entropy-based uncertainty weighting to improve generalizable deepfake image detection across generators.
Researchers propose UCF-Net, an uncertainty-aware cascaded fusion network that combines CLIP's language-aligned semantic priors with DINO's self-supervised visual-structure priors for deepfake detection. It aggregates hierarchical features across transformer depths via layer-wise expert modules and performs weighted fusion driven by entropy-derived uncertainty. The authors consolidate public deepfake datasets into a unified benchmark of roughly 4 million images plus a cross-generator set of over 8,000 faces from eight recent generators, where UCF-Net achieves the best mean AUC among evaluated methods, though zero-shot transfer remains challenging.
- Combines CLIP semantic priors with DINO visual-structure priors
- Layer-wise expert aggregation adapts multi-level encoder cues
- Unified benchmark of ~4M images plus 8K cross-generator faces
- Best mean AUC in-domain and cross-domain; zero-shot transfer stays hard
Full article170 words · extracted from huggingface.co · click to collapse
The growing realism and accessibility of manipulated and generated faces threaten the trustworthiness of digital media. To detect such forgeries, deepfake detectors based on vision foundation models have shown promising performance, but they typically rely on a single pretrained representation and are prone to overfitting to particular training distributions. To improve generalization to unseen forgeries, we propose UCF-Net, an uncertainty-aware cascaded fusion network that harnesses CLIP's language-aligned semantic priors and DINO's self-supervised visual-structure priors. UCF-Net extracts hierarchical features across Transformer depths, uses layer-wise expert aggregation to adaptively combine each encoder's multi-level cues, and performs weighted fusion of the resulting representations based on entropy-derived uncertainty. We further consolidate public deepfake datasets into a unified benchmark of approximately 4M images and construct a separate cross-generator evaluation set with over 8K face images from eight recent generators. On the unified benchmark, UCF-Net achieves the best mean AUC among the evaluated methods in both in-domain and cross-domain evaluations. On the cross-generator set, it adapts effectively with limited target-domain data, although zero-shot transfer remains challenging.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.07670