PKPS enables benchmark prediction with sparse responses
Original title: Benchmark Prediction with Sparse and Unpaired Cached Responses
PKPS represents models to predict benchmarks from sparse unpaired data.
Evaluating generative models on all benchmark queries is expensive. PKPS creates low-dimensional representations using response embeddings. It generalizes by comparing responses to similar rather than identical queries. Theoretically it is query-efficient relative to dense methods. Empirically it outperforms baselines on two large benchmark suites.
- Uses sparse and unpaired responses
- Generalizes black-box model comparisons
- Query-efficient prediction
- Outperforms baselines empirically
From the original article
1 min read at arxiv.orgEvaluating a generative model on all queries from all available benchmark tasks is prohibitively expensive. As such, models are often evaluated on different subsets of tasks and/or different subsets of queries from a given task. The resulting collection of responses and the corresponding (model, task) score matrix are sparse and unpaired.
By Hayden Helm · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.