Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
Study shows training LLMs on refusal rationales instead of boilerplate refusal statements reduces false refusals while maintaining safety performance.
The paper decomposes safety-tuning responses into a boilerplate refusal statement and an explanatory rationale, finding that refusal statements push models to rely on superficial cues and misjudge benign queries as harmful. Training solely on rationales reduces false refusals while maintaining comparable safety performance, and the benefits carry over to in-context learning configurations and remain compatible with inference-time mitigations. The results argue for precisely curated, fine-grained safety supervision datasets when aligning LLMs.
- Rationale-only training preserves safety while cutting false refusals
- Boilerplate refusal statements induce reliance on superficial risk cues
- Benefits hold in ICL settings and alongside inference-time mitigations
Full article184 words · extracted from huggingface.co · click to collapse
Striking a balance between helpfulness and safety remains a fundamental challenge in aligning large language models. To achieve this balance, models should refuse harmful queries (e.g., "How do I shoot someone?") while remaining responsive to benign inputs, even those superficially resembling harmful queries (e.g., "Where can I shoot a good photo?"). However, models often struggle to distinguish genuinely harmful queries from benign queries that contain superficially risky language, resulting in false refusals. In this paper, we address the issue by decomposing a response in the safety-tuning dataset into two distinct components: (i) a boilerplate refusal statement and (ii) a rationale explaining the refusal. Our experiments and analyses show that refusal statements impede accurate discrimination between harmful and benign queries by inducing reliance on superficial cues. In contrast, training solely on rationales reduces false refusals while maintaining a comparable level of safety performance. Rationale-Only benefits also appear in our ICL configuration and remain compatible with the evaluated inference-time mitigation methods. The results emphasize the necessity of precisely curated, fine-grained safety supervision datasets and outline directions for constructing aligned agents that better reconcile helpfulness with safety.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.04714