Copy-Paste Language Models
Copy-Paste Language Models add a lightweight copy head that improves copying, efficiency, and long-context performance.
Copy-Paste Language Models add a lightweight Copy-Paste head to a conventional language-modeling head to improve in-context copying. The authors pretrain models from 135 million to 1.7 billion parameters and report better training efficiency and downstream results across scales and backbones, with negligible overhead. They also report new operating points for long-context and retrieval settings. The paper attributes the gains partly to an alternative gradient path that mitigates the language-model head’s gradient bottleneck.
- CPLMs add a lightweight Copy-Paste head beside the language-modeling head.
- Models from 135 million to 1.7 billion parameters were pretrained.
- Gains appear across scales with negligible training and inference overhead.
- An alternate gradient path is said to ease the LM-head bottleneck.
From the original article
1 min read at huggingface.co
Transformers are intuitively considered to be efficient at copying from their inputs. We actually show that the ability of LLMs to copy in context can be improved through explicit modeling. We propose Copy-Paste Language Models (CPLMs), which add a lightweight Copy-Paste head to the conventional language modeling (LM) head.
By Nathan Godey, Yoav Artzi · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.