Bridging the Confidence Gap: Temperature Scaling for Calibrating Test-Time Prompt Tuning
CoTS temperature scaling cuts test-time prompt tuning's expected calibration error from 11.90% to 5.38% on ImageNet variants while raising accuracy.
The paper proposes CoTS, a post-hoc calibration method that applies temperature scaling to minimize the confidence gap between test-time-adapted and zero-shot predictions. A weak-strong ensemble variant, E-CoTS, further exploits multiple test-time augmentations to boost accuracy while maintaining calibration. E-CoTS reduces average expected calibration error from 11.90% to 5.38% on ImageNet variants while increasing accuracy from 60.74% to 62.95%.
- Temperature scaling aligns adapted predictions with well-calibrated zero-shot ones
- Weak-strong ensemble exploits multiple test-time augmentations
- E-CoTS cuts ECE from 11.90% to 5.38% on ImageNet variants
- Accuracy rises from 60.74% to 62.95% without calibration trade-off
Full article176 words · extracted from arxiv.org · click to collapse
Test-time prompt tuning (TPT) enables adaptation on a single test instance, achieving improved accuracy but often sacrificing calibration performance. Most existing calibration methods introduce additional regularization terms to promote dispersion across text embeddings and reduce calibration error, yet these methods often suffer from a drop in accuracy. Motivated by the well-calibrated nature of zero-shot predictions, we propose CoTS, a simple yet effective post-hoc calibration method that preserves accuracy. Specifically, CoTS applies temperature scaling to minimize the confidence gap between adapted and zero-shot predictions. To fully exploit the potential of multiple augmentations during adaptation, we introduce a weak-strong ensemble strategy that further boosts accuracy. We then apply CoTS to this ensemble, termed E-CoTS, to maintain its well-calibrated property. Extensive experiments on diverse datasets and backbones show that our approaches effectively mitigate miscalibration without compromising primary accuracy. For instance, E-CoTS reduces the average expected calibration error of TPT from 11.90% to 5.38% on ImageNet variants, while even increasing accuracy from 60.74% to 62.95%. Moreover, when integrated with existing calibration methods, E-CoTS usually enhances both accuracy and calibration simultaneously.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.17386