Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone
Neural Spectral Capacity scores architectures from their specs alone and prunes LLaMA-7B to 5.7B without calibration data.
Neural Spectral Capacity is a closed-form scalar from the singular-value spectrum of weight matrices, computable from an architecture specification alone under random initialization via the Marchenko-Pastur law. Its dynamic-programming solver, NSC-DP, maximizes capacity under resource constraints in seconds on a CPU. NSC outranks parameter and FLOP counts across seven Transformer and CNN families, discovers a WikiText-103 Transformer-XL that beats a human baseline, and prunes LLaMA-7B to a leading 5.7B model on eight commonsense tasks without calibration data, about 5900 times faster than the strongest training-free proxy.
- NSC is computed from architecture specs without instantiating the model.
- NSC-DP finds a WikiText-103 Transformer-XL beating the human baseline in two seconds.
- Prunes LLaMA-7B to 5.7B, best across eight commonsense tasks, about 5900x faster.
Full article201 words · extracted from huggingface.co · click to collapse
Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure: two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko-Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU -- a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, τ= 0.505 on pairs differing in #Params by less than 10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, about 5900x faster than the strongest training-free proxy baseline.
Text extracted automatically; images, tables and formatting may be missing. Original: https://huggingface.co/papers/2609.23087