FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models
FactorEngram adds factorized n-gram memory with basis-level gating to language models.
FactorEngram replaces monolithic n-gram embeddings with sparsity-regularized coefficients over a shared dictionary of basis vectors, which the backbone hidden state gates individually. It covers both single tokens and multi-token n-grams and studies where the memory branch should be inserted. On 340M- and 1B-parameter Transformer backbones it improves language modeling and downstream tasks. Ablations identify insertion before the attention sublayer in the middle layers as an effective configuration.
- Shared basis vectors let related n-grams reuse components
- Context gates each memory coefficient individually
- Improvements reported on 340M and 1B Transformers
- Best placement is before attention in middle layers
From the original article
1 min read at huggingface.co
Lookup-based memory has been a promising way to scale the parameters of large language models (LLMs). It retrieves learned representations of local token patterns, such as n-grams, instead of reconstructing them through successive layers of computation. However, existing designs such as Engram treat each retrieved embedding as a monolithic unit.
By Bowen Yang, Jingbo Zhou, Qinghong Miao · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.