Hugging Face trending modelspublished Aug 26, 12:29 UTC (20d ago)ingested Sep 8, 14:03 UTCunsloth1

unsloth/Qwen3.8-Flash-Next-GGUF — new model trending #21 on Hugging Face

AI summary · glm-5.3-flash

Qwen released Qwen3.8-Flash-Next, an experimental 125B-parameter open-weight MoE previewing the Qwen4 architecture, with Unsloth shipping optimized GGUF quants.

Qwen released Qwen3.8-Flash-Next, an experimental open-weight preview of the architecture planned to underpin Qwen4. The model has 125B parameters with 6B activated, 512 experts (10 routed plus 1 shared), Qwen Sparse Attention (QSA), Gated DeltaNet, Gated Residual, and n-gram embeddings, with 262,144-token native context extendable to 1,000,000 tokens. Unsloth provides Dynamic 3.0 GGUF quantizations, and multi-token prediction (MTP) delivers 1.3-1.7x faster inference via llama.cpp or Unsloth Desktop.

125B-parameter MoE with 6B activated per token
Qwen Sparse Attention cuts long-context latency
Native 262K context, extensible to 1M tokens
Unsloth GGUF quants and MTP speed inference 1.3-1.7x

Read the original

VendorsQwen Unsloth

ProductsUnsloth Desktop llama.cpp

OrganizationsQwen Unsloth

AI modelsQwen3.8-Flash-Next

Tagsqwen qwen3.8-flash-next open-weights moe sparse-attention gguf unsloth long-context

Full article1,313 words · extracted from huggingface.co · click to collapse

# Read our How to [Run Qwen3.8-Flash-Next Guide!](https://unsloth.ai/docs/models/qwen3.8-next)

<div>

<em><a href="https://unsloth.ai/docs/basics/dynamic-3.0-ggufs">Unsloth Dynamic 3.0</a> achieves superior accuracy & outperforms other leading quants.</em>

</p>

</a>

</a>

</a>

</div>

<li>MTP is now available for 1.3-1.7x Faster inference in Unsloth. <a href="https://unsloth.ai/docs/models/qwen3.8-next#mtp-guide">Read Guide</a></li>

<li>To run, please use llama.cpp or use our <a href="https://unsloth.ai/docs/desktop">Unsloth Desktop</a> app.</li>

<li>See below for Qwen3.8-Flash-Next run in Unsloth Desktop with thinking controls:</li>

</div>

---

# Qwen3.8-Flash-Next

As the frontier of foundation models pushes toward ever-larger parameter counts and ever-longer context windows, the question is no longer just how much we can scale, but how efficiently we can do so. Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation. Today, we are sharing a concrete step in that direction: Qwen3.8-Flash-Next.

![Qwen3.8-Flash-Next Architecture](https://qianwen-res.oss-accelerate.aliyuncs.com/Qwen3.8-Flash-Next/architecture.png)

This experimental preview of the architecture that will underpin Qwen4 is built around a fundamental rethinking of how the core components of modern large language models (LLMs) interact at scale.

## Highlights

The first open-weight release under this architecture is Qwen3.8-Flash-Next, which introduces:

- **Hybrid Attention with QSA**: The Gated DeltaNet and Gated Attention pairing has been reworked into Gated DeltaNet and Qwen Sparse Attention (QSA). Rather than selecting individual tokens for processing, QSA operates at the micro-block level. This cuts long-context latency significantly, a critical gain as agentic workloads increasingly dominate real-world usage.

- **Gated Residual**: Residual streams with normalization are what make deep LLM training manageable. Gated Residual modulates information flowing through widened residual streams via an element-wise, data-dependent read gate and a per-branch scalar write gate. This brings finer-grained expressiveness across layers while preserving training stability and keeping inference overhead low.

- **N-gram Embedding**: Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts (MoE). By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality.

- **Tailored Training Recipe**: The Muon and AdamW optimizers are applied to specific weight categories to maximize efficiency. Guided by refitted scaling laws, we eliminate traditional batch-size warmups and start directly at the target batch size, substantially reducing total optimizer steps while safely supporting larger learning rates for robust convergence.

For more details, please refer to our blog post [Qwen3.8-Flash-Next](https://qwen.ai/blog?id=qwen3.8-flash-next) and [the technical report](https://github.com/QwenLM/Qwen3.8-Flash-Next/blob/main/tech_report.pdf).

We are excited to embark on this next chapter with you and welcome your feedback as we build what comes next.

## Model Overview

- Type: Causal Language Model with Vision Encoder

- Training Stage: Pre-training & Post-training

- Language Model

- Number of Parameters: 125B with 6B activated, plus 51B n-gram embedding and 4B MTP

- Hidden Dimension: 2560

- Token Embedding: 248320 (Padded)

- N-gram Embedding: 20,000,000 (bigrams/trigrams at layer 2)

- Number of Layers: 48

- Hidden Layout: 12 × (3 × (Gated DeltaNet → MoE) → 1 × (Qwen Sparse Attention → MoE))

- Gated DeltaNet:

- Number of Linear Attention Heads: 48 for V and 16 for QK

- Head Dimension: 128

- Qwen Sparse Attention:

- Number of Attention Heads: 24 for Q and 2 for KV

- Head Dimension: 256

- Rotary Position Embedding Dimension: 64

- Indexer Structure: MQA with 4 Query Heads and 1 Shared Key Head

- Indexer Head Dimension: 128

- Budget: 512 blocks or 2048 tokens

- Mixture Of Experts

- Number of Experts: 512

- Number of Activated Experts: 10 Routed + 1 Shared

- Expert Intermediate Dimension: 640

- Gated Residual:

- Number of Branches: 4

- Bottleneck Rank: 320

- LM Output: 248320 (Padded)

- MTP: 1 layer, trained with multi-steps

- Context Length: 262,144 natively and extensible up to 1,000,000 tokens.

## Benchmark Results

<style>

.vl-table th{font-size:15px!important;line-height:1.2}

.vl-table td:not(.benchmark-cell):not([colspan]){font-size:15px;line-height:1.2;vertical-align:middle}

.vl-table .benchmark-cell{padding:12px 10px 12px 18px!important;vertical-align:middle}

.vl-table .benchmark-capability{font-size:15px;font-weight:600;line-height:1.22;color:#171717}

.vl-table .benchmark-name{margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B}

.vl-table .metric-stack{display:flex;flex-direction:column;gap:7px;padding:3px 0}

.vl-table .metric-label{font-size:10px;font-weight:400;line-height:1.1;color:#777}

.vl-table .metric-value{margin-top:2px;font-size:15px;line-height:1.15;color:#171717}

.vl-table .metric-pair{white-space:nowrap}

.vl-table .metric-sep{color:#9A9A9A;padding:0 3px}

</style>

### Language

<th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #0A2EFE;color:#0A2EFE"></th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;background:rgba(10, 46, 254, 0.08);">Qwen3.8-Flash-Next</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Qwen3.8-27B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Qwen3.7-Plus</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">DeepSeek-V4-Flash-0731</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #0A2EFE;color:#0A2EFE;font-size: 14px;width:14.00%;">Claude-Opus-4.6 (Max)</th></tr></thead>

<tbody>

<tr>

</tr>

<tr>

</tr>

<tr>

</tr>

<tr><td colspan="6" style="padding:8px 12px;font-weight:600;color:#0A2EFE;border-bottom:1px solid rgba(10, 46, 254, 0.2);background:#D6DAFC">Coding</td></tr>

<tr>

<td class="benchmark-cell" style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);"><div class="benchmark-capability" style="font-size:15px;font-weight:600;line-height:1.22;color:#171717">Agentic coding</div><div class="benchmark-name" style="margin-top:4px;font-size:11px;font-weight:400;line-height:1.2;color:#6B6B6B">DeepSWE 1.1</div></td>

</tr>

<tr>

Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF