Nearly Tight Rademacher Bounds for Sparsely Activated Neural Networks
Theory paper derives nearly tight Rademacher complexity bounds for sparsely activated one-hidden-layer ReLU networks.
Building on Awasthi et al. (COLT 2024), the authors bound statistical complexity for networks where each input activates at most k of s hidden units. A support-preserving cover and normalized chaining argument remove the explicit dimension factor, with matching lower bounds up to logarithms. They also derive agnostic minimax excess-risk bounds of order min{1, sqrt(s/(km))} for a normalized bounded loss and show bias bounds comparable to WR restore worst-case rates even on domains where sparsity holds globally.
- Upper bound CWR*min{k, sqrt(sk/m)}*log^1.5(2m)+kB/sqrt(m) for width-s networks with k active units
- Lower bounds on i.i.d. marginals match the upper bound up to logarithmic factors
- Zero-bias ball-sparse networks retain complexity O(kWR/sqrt(m)); large biases restore worst-case rate
From the original article
1 min read at arxiv.orgAn input may activate few hidden units even when different inputs collectively use an entire network. We study the statistical complexity of this input-dependent sparsity in the one-hidden-layer ReLU model of Awasthi et al. (COLT 2024).
By Xiaoyu Li · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.