HyQuant: Hybrid-Precision Quantization for LLM Attention
HyQuant keeps most LLM attention states low-bit while preserving vertical-line tokens and local windows in high precision, maintaining near-lossless accuracy.
HyQuant is a hybrid-precision quantization framework for LLM attention that quantizes most attention states to low bits while keeping accuracy-critical vertical-line tokens and local-window states in full precision, selected via lightweight attention-pattern signals. In the prefill stage it uses a hybrid-precision attention operator, and in the decode stage it applies the same principle to KV-cache compression with fused dequantization and attention computation. Across diverse tasks, models, and datasets it maintains nearly lossless accuracy; code is available on GitHub.
- Hybrid design avoids reliance on outlier smoothing for low-bit attention quantization
- Vertical-line-aware signals identify accuracy-critical tokens to keep in high precision
- Applies to both prefill attention and decode-stage KV-cache compression
- Fuses KV dequantization with attention computation for memory and hardware efficiency
- Near-lossless accuracy reported across tasks, models, and datasets
Full article194 words · extracted from huggingface.co · click to collapse
Quantization has been widely adopted in LLM training and inference to reduce cost and improve efficiency. However, low-bit quantization of the attention module often introduces large errors at very low bit-widths, causing performance degradation. Existing methods mainly rely on smoothing techniques to handle outliers, while we propose a hybrid quantization design to better balance accuracy and efficiency. Specifically, we propose HyQuant, an efficient hybrid quantization framework for LLM attention. HyQuant quantizes most attention states into low-bit formats while retaining a small set of vertical-line tokens and local-window states in high precision. These accuracy-critical regions are selected using lightweight vertical-line-aware attention-pattern signals, reducing quantization error with limited overhead. In the Prefill stage, HyQuant uses a hybrid-precision quantized attention operator that preserves vertical-line tokens and a local sliding window in full precision while quantizing the remaining context. In the Decode stage, HyQuant applies the same principle to KV-cache compression and fuses KV dequantization with attention computation to improve memory and hardware efficiency. Across diverse tasks, models, and datasets, HyQuant maintains nearly lossless accuracy with an extremely simple design, demonstrating the efficiency and practical feasibility of hybrid quantization for LLM attention. Code is available at: https://github.com/jerrysfls/HyQuant .
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2608.27875