MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
MARGIN calibrates multi-model confidence at runtime and lifts answer-selection accuracy by up to 14 points.
MARGIN learns model-specific confidence corrections from recent answer outcomes without retraining models or using a held-out calibration set. Corrected scores weight answers from heterogeneous foundation models during collective decisions. On an 18-model pool, including BigCodeBench where mean confidence was negatively related to accuracy, MARGIN beat most online baselines on post-shift expected calibration error. In code-generation coordination, answer-selection accuracy rose by 4.3 and 14.0 percentage points on two of three benchmarks versus uncalibrated confidence weighting.
- Runtime calibration needs no retraining or held-out calibration set.
- Evaluation used an 18-model pool and a nine-model shift subset.
- On BigCodeBench, stated confidence was negatively related to accuracy.
- Answer-selection accuracy rose 4.3 and 14.0 points on two benchmarks.
- Products
- BigCodeBench
From the original article
1 min read at huggingface.co
When a coordinator compares answers from heterogeneous foundation models, self-reported confidence may have different meanings across responders and changing workloads.
By Joss Armstrong · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.