Harnessing CLIP and DINO: An Uncertainty-Aware Cascaded Fusion Network for Generalizable Deepfake Image Detection
UCF-Net fuses CLIP and DINO features with entropy-based uncertainty weighting to improve generalizable deepfake image detection across generators.
Researchers propose UCF-Net, an uncertainty-aware cascaded fusion network that combines CLIP's language-aligned semantic priors with DINO's self-supervised visual-structure priors for deepfake detection. It aggregates hierarchical features across transformer depths via layer-wise expert modules and performs weighted fusion driven by entropy-derived uncertainty. The authors consolidate public deepfake datasets into a unified benchmark of roughly 4 million images plus a cross-generator set of over 8,000 faces from eight recent generators, where UCF-Net achieves the best mean AUC among evaluated methods, though zero-shot transfer remains challenging.
- Combines CLIP semantic priors with DINO visual-structure priors
- Layer-wise expert aggregation adapts multi-level encoder cues
- Unified benchmark of ~4M images plus 8K cross-generator faces
- Best mean AUC in-domain and cross-domain; zero-shot transfer stays hard
From the original article
1 min read at huggingface.co
The growing realism and accessibility of manipulated and generated faces threaten the trustworthiness of digital media. To detect such forgeries, deepfake detectors based on vision foundation models have shown promising performance, but they typically rely on a single pretrained representation and are prone to overfitting to particular training distributions.
By Xuechao Zou, Yi Zhou, Kai Li · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.