Partially trainable networks show a sharp approximation phase transition
Original title: Phase Transition from Random Features to Fully Trainable Networks
For spherical biased-ReLU features at d=5 and τ=10, about 4.1% trainable width captures 90% of the bound reduction.
The paper quantifies how two-layer networks move from fixed random features to fully trainable representations. Under polynomial spectral decay with α>1 and leverage compatibility, it proves a squared L2 rate that depends on width m and trainable proportion P. The bound holds uniformly over the unit ball of an RKHS with probability at least 1−δ. Experiments, including networks with up to 15 layers, show the same pattern in training error: fast early gains, then diminishing returns.
- κ_P mixes fixed and trainable sampling mismatches κ0* and κ1*.
- For biased-ReLU features, α equals (d+2)/(d−1) and κ1* is 1 under ridge leverage sampling.
- κ0* is 4√2 τ(d+1)/d for initialization scale τ≥1.
- A three-layer example shows a few trainable first-layer neurons using fixed downstream blocks.
- Greater initialization mismatch makes the transition sharper.
From the original article
1 min read at arxiv.orgConstructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations.
By Xin He · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.