Ensemble Methodology Estimates LLM Code Summary Accuracy Without References
Original title: Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
Inter-model agreement and LLM judges enable reference-free correctness estimation for LLM-generated COBOL summaries.
The paper presents an approach for estimating correctness of LLM-generated COBOL summaries without human references. The core idea is that agreement between models indicates truthfulness. Controlled generation from diverse models is combined with an LLM evaluator that performs pairwise semantic comparisons. A calibrated aggregation rule integrates supportive evidence while penalizing contradictions.
- No access to human-authored references needed
- Uses controlled summary generation from multiple models
- LLM acts as judge for pairwise assessments
- Calibrated aggregation combines evidence and penalties
From the original article
1 min read at arxiv.orgThis paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics.
By Marco Colombo · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.