Theory separates locality from weight sharing in convolutional learning
Original title: The role of locality and weight sharing in convolutional neural networks
With uncorrelated inputs a convolutional student learns exactly P times faster, but spatial correlation can erase that edge.
A teacher-student analysis compares convolutional and locally connected students trained by online stochastic gradient descent on P positions with correlation c. Exact ODEs describe the dynamics in the large feature-dimension limit. For uncorrelated positions the trajectories match and the convolutional student is exactly P times faster; matching the teacher's kernel size recovers it from O_D(D) samples, while a larger kernel needs O_D(D log D). For c greater than 0, correlations raise the convolutional student's asymptotic error, and which student wins depends on whether the learning rate is fixed or tuned to the horizon.
- Locality changes sample complexity: matching kernel size needs O_D(D) samples.
- A larger kernel requires O_D(D log D) samples.
- The locally connected student's asymptotic error does not depend on c.
- With a horizon-tuned learning rate, the locally connected student wins early and the convolutional student late.
From the original article
1 min read at arxiv.orgThe success of convolutional neural networks is often attributed to locality and weight sharing. However, how these two properties separately shape learning, and how their effects depend on the spatial correlations of natural images, remain poorly understood. We address these questions in a teacher-student model with a convolutional teacher acting on $P$ input positions with correlation $c$ between positions.
By Francesco Mori · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.