ReLU networks with one hidden layer are weakly closed
Original title: On the (non-)closedness of the set of ReLU networks
One-hidden-layer ReLU networks are weakly closed enabling energy minimization in Deep Ritz and Sobolev training.
The set of ReLU networks with one hidden layer at most N neurons is weakly closed in W^{1,p}(Omega) for bounded open sets. This covers Deep Ritz with convex integrands Sobolev training and Lipschitz learning. As weight decay approaches zero minimizers converge to minimal norm solutions. Two-hidden-layer networks are not closed in W^{1,p}(Omega) or C(Omega) disproving a prior conjecture.
- Weakly closed in W^{1,p}(Omega)
- Covers Deep Ritz Sobolev and Lipschitz methods
- Minimizers converge to minimal norm as decay to zero
- Two-hidden-layer sets not closed
- Disproves Petersen Raslan Voigtlaender conjecture
From the original article
1 min read at arxiv.orgWe prove that the set $\mathcal{N}_N$ of ReLU networks with one hidden layer and at most $N\in\mathbb N$ neurons is weakly closed in $W^{1,p}(Ω)$, for a bounded connected open set $Ω\subset\mathbb R^d$. Consequently, every coercive, weakly lower semicontinuous energy on this space attains its minimum over $\mathcal{N}_N$.
By Eloi Martinet · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.