TWIG stores neural weights as shared paths of discrete decisions
Original title: Every Weight Is a Path of Shared Discrete Decisions
On five LLM weight matrices, a flat power-of-two TWIG needs 9-15% fewer additions than GPTQ at equal error.
TWIG represents each neural-network weight as the end of a tree of small discrete decisions: a sign, a power-of-two shift, a zero gate, and an index into a shared table. Shared upper decisions are stored and applied once, so pruning, quantization, low-rank factorization, codebooks, and weight sharing become edits of one object. Power-of-two decisions turn weight layers into circuits made mostly of wires and adders. Choosing the decisions is cast as a higher-order Boltzmann machine ground state.
Unconfirmed: The large language model whose five weight matrices were measured is not named.
- The paper says any trained network can be written as a TWIG.
- Comparison uses five weight matrices from an unnamed large language model.
- At 3- and 4-bit accuracy, typical matrices need 9-15% fewer additions than GPTQ.
- Matrices with strong outliers need 20-55% fewer additions, with no group scales.
From the original article
1 min read at huggingface.co
We introduce TWIG (Tree-structured Weights with Inherited Gates), a representation in which every weight of a neural network is the end of a path of small discrete decisions: a sign, a shift by a power of two, a zero gate, an index into a shared table. Weights that belong together share the upper part of their paths, and each decision is inherited by every weight below it, so it is stored once and applied once.
By Ammar Ali, Stamatis Lefkimmiatis · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.