MiST: Mid-Training LLMs for Cybersecurity
MiST introduces 8B and 32B cybersecurity-specialized LLMs that outperform Qwen baselines by up to 13.1 points on public security benchmarks.
MiST (Mid-trained Security Transformer) applies mid-training as an intermediate adaptation stage, converting an expert-vetted seed corpus into high-quality synthetic domain data rather than continual pretraining on raw text. The 8B and 32B checkpoints improve mean cybersecurity accuracy by +13.1 and +8.6 absolute points over Qwen baselines (+27.0% and +15.8% relative). Ablations show gains arise in mid-training and supervised fine-tuning, and MiST provides stronger initialization for downstream fine-tuning and reinforcement learning.
- Mid-training converts curated expert-vetted corpus into synthetic domain training data
- 8B model gains +13.1 points (+27.0% relative) over Qwen baseline
- 32B model gains +8.6 points (+15.8% relative) over Qwen baseline
- Checkpoints strengthen initialization for task-specific fine-tuning and RL
From the original article
1 min read at arxiv.orgCybersecurity combines high-stakes analysis with complex technical language, making it an impactful and challenging domain for LLMs. We present MiST (Mid-trained Security Transformer), a suite of 8B and 32B models that achieve strong performance on public cybersecurity benchmarks. We use mid-training as an intermediate adaptation stage between general pre-training and cybersecurity training.
By Oded Ovadia · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.