OPTIC compresses Gemma 4 E4B per-layer embeddings up to 1,002×
Original title: Extreme Compression of Per-Layer Embeddings via Cross-Layer Latent Vector Quantization
Authors report 97.8% of uncompressed performance at about 100× PLE compression, versus 66.3% for product quantization.
The paper introduces OPTIC, which stores one quantized code tuple per token and decodes it at every layer with a layer-specific affine decoder fitted to the PLE output path. Per-layer embeddings are 35.2% of Gemma 4 E4B parameters. Authors report OPTIC retains 97.8%, 90.5%, and 84.1% of uncompressed BF16 E4B performance at about 100×, 594×, and 1,002× PLE compression, and 96.9% on official W4A16 E4B at about 100×. At that rate they report 29.7% less GPU memory and at most 0.40% slower inference.
Unconfirmed: Retention, memory, and speed figures are the authors' reported results, not independently verified.
- PLE accounts for 35.2% of Gemma 4 E4B parameters.
- Product quantization, the strongest of six baselines, retains 66.3% at about 100×.
- On W4A16 E4B, OPTIC retains 96.9% at about 100× PLE compression.
- BF16 E4B uses 29.7% less GPU memory at the same KV cache size.
- Inference is at most 0.40% slower than uncompressed PLE.
From the original article
1 min read at huggingface.co
Scaling embedding capacity alongside Transformer computation is an emerging trend in language models. This trend makes high compression rates increasingly useful. Hence, we study extreme compression of Per-Layer Embeddings (PLE), which follow this trend and account for 35.
By Jiacheng Liu, Yaxin Luo, Xiaohan Zhao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.