Everything in Moderation: Per-Domain Coverage Optima and Alignment-Resistant Domain Gaps in Multi-Domain Mid-Training
Controlled mid-training experiments on Qwen3-8B-Base find each domain has a 10-40% coverage optimum and domain gaps survive alignment SFT.
Using Qwen3-8B-Base (with a 4B replication) across five semantically rule-disjoint KOR-Bench domains, the authors train 30 data allocations spanning the five-domain simplex at five seeds each. All five domains show interior optima in the moderate 10-40% coverage band, and domain gaps persist after a fixed-budget compensatory SFT pass, which raises 116/120 cells yet bridges 0/240 pairs at a 5% threshold. Zero coverage collapses mid-training-only accuracy, though a FineWeb-Edu-only control shows the collapse is partly generic drift. The results argue mid-training data composition requires principled design rather than reliance on later alignment.
- 30 allocations across five KOR-Bench domains, five seeds each
- Moderate 10-40% per-domain coverage is best for all five domains
- Compensatory SFT bridges 0/240 domain-gap pairs at a 5% threshold
- Zero coverage collapses mid-training accuracy, partly from generic drift
- Organizations
- KOR-BenchFineWeb-Edu
- AI models
- Qwen3-8B-Base
From the original article
1 min read at arxiv.orgMid-training, the stage between pre-training and alignment, is where a model's per-domain data composition is typically set by data availability rather than principled design. We ask what that decision buys, and whether a later alignment pass can undo it.
By Yunpeng Xu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.