Paper estimates ReLU-net mutual information from routing over linear regions
Original title: Intrinsic Information-Theoretic Analysis of ReLU Nets
The routing term uses a plug-in estimator and, when well sampled, approximates a lower bound on joint information.
The paper argues that continuous piecewise-linear networks, including ReLU nets, can be analysed without high-dimensional mutual-information estimators whose bias swamps the signal. Each input gets a discrete region label from the network's partition into convex linear regions, plus the continuous representation and the label. Mutual information is decomposed into geometric terms, with routing information as the dominant estimable term. A functional-equivalence quotient collapses regions that implement the same linear operator when the partition saturates.
- The routing estimator agrees strongly with three established mutual-information baselines.
- The quotient has one bounded dimensionless parameter, reported as a compression ratio.
- The quotient recovers estimates near the boundary of well-sampled and under-sampled regimes.
- The method uses the network's own piecewise-linear partition rather than external discretisation.
From the original article
1 min read at arxiv.orgMutual information (MI) is fundamental to representation-learning analysis in deep networks, but in high dimensions MI estimators depend on hyperparameters whose bias often dominates the signal.
By Johan Mylius-Kroken · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.