Unifying Conformal Language Tasks with In-Context Ensembles
Researchers propose Conformal Relevance, which builds conformal score functions via in-context example curation and ensembling to improve conciseness across seven NLP tasks.
The paper targets NLP tasks like summarization and extractive QA that reduce to retrieving content under coverage and conciseness constraints. Conformal Relevance replaces hand-engineered LLM scoring prompts with curated in-context examples and ensembles, maintaining coverage guarantees while improving conciseness with minimal manual input. The authors demonstrate the framework on seven NLP tasks and contribute theory, including a complementarity condition for when ensembling improves worst-case sentence scores and a saturation bound on ensemble gains.