Cross-modal steering vectors transfer between text and image generation
Original title: Cross-modal Steering Vectors Exist Across Both Image and Text Generation
In unified multimodal models, vectors found on one modality steer the other zero-shot, including concepts unique to one side.
The paper tests whether concepts are shared across modalities by finding steering vectors that change generation for one token type and applying them zero-shot to the other. In unified multimodal models with shared language and image parameters, vectors transfer both text-to-image and image-to-text for concepts such as emotion and object size. Even modality-specific ideas, such as formal versus colloquial language, often produce a consistent effect in the other modality, and transferable vectors tend to point in similar directions despite a modality gap.
- Steering is treated as a causal test, not mere representational similarity.
- Transfer works in both directions across diverse steered concepts.
- Formal language steering often yields formal clothing and design in images.
- Strongly transferable vectors align geometrically for the same concept.
From the original article
1 min read at huggingface.co
Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality.
By Donghao Zeng, Nishant Subramani, Tamar Rott Shaham · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.