DiaVLo: Diagnosing Behaviours of Vision-Language Models
DiaVLo diagnoses vision-language model behaviours, comparing desired and observed actions and estimating influential concepts.
DiaVLo is a diagnostic framework for vision-language models that uses human curation and the models' own generation to specify desired and observed behaviours and expose misalignment. It also estimates which concepts most strongly steer those behaviours. Evaluations on several open-source VLMs under classification and generation show that the resulting behaviour labels correlate with measured performance and add context to those scores. The method surfaces clearly aligned and misaligned behaviours plus patterns in how models perceive, organize, and prioritize concepts.
- Combines human curation with model generation to specify desired and observed behaviours.
- Provides causal estimates of concepts that most influence VLM behaviour.
- Labels correlate with performance in classification and generation settings.
- Experiments surface both aligned and misaligned behaviours in open-source VLMs.
Full article131 words · extracted from arxiv.org · click to collapse
Vision-language models (VLMs) rely on storing and transferring appropriate information across their sub-components. Verifying that the VLMs exhibit desired behaviours, while avoiding harmful ones, is central to their reliable deployment. Yet, methods that identify VLM behaviours remain scarce. We present DiaVLo, a diagnostic framework that leverages human curation and VLMs' generation capabilities to construct specifications of desired and observed VLM behaviours, surfacing potential misalignments. Beyond this, DiaVLo also provides causal estimates to identify the most influential concepts steering VLM behaviours. We evaluate DiaVLo on several open-source VLMs under both classification and generation conditions. Our experiments show that DiaVLo produces behaviour labels that correlate with model performance and provide context for measured performance. DiaVLo surfaced behaviours that are clearly aligned and misaligned, alongside patterns in how VLMs perceive, organise, and prioritise concepts.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.22008