Unifying Conformal Language Tasks with In-Context Ensembles
Researchers propose Conformal Relevance, which builds conformal score functions via in-context example curation and ensembling to improve conciseness across seven NLP tasks.
The paper targets NLP tasks like summarization and extractive QA that reduce to retrieving content under coverage and conciseness constraints. Conformal Relevance replaces hand-engineered LLM scoring prompts with curated in-context examples and ensembles, maintaining coverage guarantees while improving conciseness with minimal manual input. The authors demonstrate the framework on seven NLP tasks and contribute theory, including a complementarity condition for when ensembling improves worst-case sentence scores and a saturation bound on ensemble gains.
- Replaces manual prompt engineering in conformal scoring with curated in-context ensembles
- Evaluated across seven NLP tasks with coverage guarantees retained
- Provides complementarity condition and saturation bound for ensembled conformal scores
From the original article
1 min read at huggingface.co
Many NLP tasks, such as summarization and extractive question answering, reduce to retrieving relevant content from documents under two constraints: coverage, retaining enough pertinent information to achieve some goal, and conciseness, removing as much irrelevant information as possible.
By Xiao Shi Huang, Chen-Yuan Lin, Bruce Kuwahara · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.