MiST: Mid-Training LLMs for Cybersecurity
MiST introduces 8B and 32B cybersecurity-specialized LLMs that outperform Qwen baselines by up to 13.1 points on public security benchmarks.
MiST (Mid-trained Security Transformer) applies mid-training as an intermediate adaptation stage, converting an expert-vetted seed corpus into high-quality synthetic domain data rather than continual pretraining on raw text. The 8B and 32B checkpoints improve mean cybersecurity accuracy by +13.1 and +8.6 absolute points over Qwen baselines (+27.0% and +15.8% relative). Ablations show gains arise in mid-training and supervised fine-tuning, and MiST provides stronger initialization for downstream fine-tuning and reinforcement learning.
- Mid-training converts curated expert-vetted corpus into synthetic domain training data
- 8B model gains +13.1 points (+27.0% relative) over Qwen baseline
- 32B model gains +8.6 points (+15.8% relative) over Qwen baseline
- Checkpoints strengthen initialization for task-specific fine-tuning and RL
Full article155 words · extracted from arxiv.org · click to collapse
Cybersecurity combines high-stakes analysis with complex technical language, making it an impactful and challenging domain for LLMs. We present MiST (Mid-trained Security Transformer), a suite of 8B and 32B models that achieve strong performance on public cybersecurity benchmarks. We use mid-training as an intermediate adaptation stage between general pre-training and cybersecurity training. Rather than performing continual pre-training over large volumes of raw domain text, we curate a compact, expert-vetted seed corpus, and transform it into high-quality domain-specific synthetic training data. The final MiST checkpoints improve mean cybersecurity accuracy by +13.1 and +8.6 absolute percentage points over the corresponding Qwen baselines for 8B and 32B, respectively, corresponding to relative gains of +27.0% and +15.8%. Ablation results further show that these cybersecurity gains arise in the mid-training and supervised fine-tuning stages through a combination of the synthetic data generation flows. Furthermore, we show that MiST provides a stronger initialization for downstream task-specific fine-tuning adaptation and reinforcement learning.
Text extracted automatically; images, tables and formatting may be missing. Original: https://arxiv.org/abs/2609.18496