Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Models
Researchers introduce KoNA, a benchmark exposing vision-language models' failures at selective non-compliance, plus fine-tuning that improves refusal and abstention accuracy.
KoNA is a benchmark for evaluating selective non-compliance in vision-language models across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility and Safety. It tests both query-level and component-level non-compliance using paired single and compound queries, and evaluations across diverse VLMs show models often fail to refuse, correct or abstain appropriately, with failures worsening on compound queries. Fine-tuning VLMs on KoNA examples substantially improves non-compliance accuracy while largely maintaining performance on fully answerable tasks.