DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek releases V4.1-Flash, a 552B-parameter multimodal MoE with 1M-token context and 890-byte-per-token KV cache for cheaper agentic workloads; weights on Hugging Face.
DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, activating 16B parameters per token during decode and 8B during prefill, supporting contexts up to one million tokens. It combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching, reducing global KV cache to 890 bytes per token (about one-quarter of DeepSeek-V4-Flash), while SWA Bounded Replay cuts the persistent SSD/host-memory cache to roughly one-eighth. Pretrained on 45T multimodal tokens, it outperforms the baseline across text and multimodal agentic scenarios, with checkpoints released on Hugging Face.
- 552B-parameter multimodal MoE activating 16B parameters per decode token and 8B during prefill.
- CSA2 cross-layer KV reuse plus FP4 caching cuts KV footprint to 890 bytes per token.
- SWA Bounded Replay reduces persistent SSD/host-memory KV cache to roughly one-eighth of DeepSeek-V4-Flash.
- Pretrained on 45T multimodal tokens; checkpoints released on Hugging Face under deepseek-ai.
- Supports contexts up to one million tokens, targeting input-heavy agentic workloads.
Full article250 words · extracted from huggingface.co · click to collapse
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.19969