AttentionSwitch: Joint Learning for LLM Jailbreak Defense
AttentionSwitch jointly optimizes sparse attention head selection and query-conditioned scaling to defend LLMs against jailbreaks without catastrophic forgetting.
Stories per day · 30 days
2 RSS
Relateddefense1document-understanding1gated-associative-lookup1icl-circuit1in-context-learning1jailbreak1mllm1qwen1retrieval1transformer-circuits1
6 stories, newest first · one card per story
Stories tagged #attention-heads
Circuit for In-Context Learning in Language Models
The model implements a gated associative lookup using key-value memory over label-token positions and attention heads for few-shot sentence classification.
Retrieval heads stay sparse and shared across MLLM modalities
Ranking training on those heads raises Qwen3-VL-8B page Recall@1 on MMDocIR by 6.03 points.
Task Context Shapes Component Roles in Language Models
Task context changes which facts shared language-model heads use and how they influence later computation.
A mechanistic study of language model introspection
Researchers identify attention-head circuits that let language models detect and localize hidden activation interventions.
MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
Researchers propose HEAL, a method that reduces multimodal LLM hallucinations by calibrating information distribution drift in synergy attention heads.