Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone
Neural Spectral Capacity scores architectures from their specs alone and prunes LLaMA-7B to 5.7B without calibration data.
Neural Spectral Capacity is a closed-form scalar from the singular-value spectrum of weight matrices, computable from an architecture specification alone under random initialization via the Marchenko-Pastur law. Its dynamic-programming solver, NSC-DP, maximizes capacity under resource constraints in seconds on a CPU. NSC outranks parameter and FLOP counts across seven Transformer and CNN families, discovers a WikiText-103 Transformer-XL that beats a human baseline, and prunes LLaMA-7B to a leading 5.7B model on eight commonsense tasks without calibration data, about 5900 times faster than the strongest training-free proxy.
- NSC is computed from architecture specs without instantiating the model.
- NSC-DP finds a WikiText-103 Transformer-XL beating the human baseline in two seconds.
- Prunes LLaMA-7B to 5.7B, best across eight commonsense tasks, about 5900x faster.
- Tags
- neural-spectral-capacityarchitecture-searchtransformerpruningllamatraining-freemodel-compression
- Vendors
- Meta
- Products
- LLaMA-7BTransformer-XL
- AI models
- LLaMA-7B
From the original article
1 min read at huggingface.co
Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently.
By Chenyu Zhu, Ruoyu Zhao, Zhichao Lu · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.