BYOD converts AR models to diffusion LLMs via LoRA
Original title: BYOD: Build Your Own DLM - Efficient Autoregressive-to-Diffusion Language Model Conversion
BYOD enables parameter-efficient conversion of Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B in 3.1-4.0 hours using 4.1-5.8% LoRA adapters.
BYOD provides a LoRA-only recipe for converting autoregressive language models to diffusion language models without full retraining. Adapters comprise 4.1-5.8% of parameters while freezing all base-model weights. The method was tested on Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B models. BYOD-Gemma reaches 50.6% average accuracy on a nine-task evaluation, close to LLaDA 8B Instruct at 51.9%. Open-generation quality at low network function evaluation budgets remains the main limitation.
- 4.1-5.8% LoRA adapters for full conversions
- Conversions complete in 3.1-4.0 hours on one GPU with at most 102.4M tokens
- All base-model parameters frozen
- BYOD-Gemma leads at 50.6% average accuracy
- Perplexity gap persists at low NFE budgets
From the original article
1 min read at huggingface.co
Unlike autoregressive (AR) models, diffusion language models (DLMs) permit parallel token updates, flexible generation, and bidirectional attention, but leading systems rely on costly pretraining or full-model fine-tuning. We present Build Your Own DLM (BYOD), a generalizable LoRA-only recipe for parameter-efficient AR-to-DLM conversion.
By Ruurd J. A. Kuiper, Robert A. Bagheri · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.