Learning by Compression: Risk Estimates Attain the Law-of-Large-Numbers Rate
Compression-based learning risk bounds match the law-of-large-numbers rate without a logarithmic penalty.
The paper derives risk bounds for learning rules based on preferent compression, where complexity k is the size of the compression plus training examples on which the reconstructed hypothesis fails. Earlier bounds sat an O(sqrt(ln N/N)) margin above k/N. For any fixed confidence 1-δ, the authors prove the risk is at most k/N + O(1/sqrt(N)), uniformly in k, matching the law-of-large-numbers rate. They show k/N is necessary and identify regimes where empirical risk converges more slowly or loses consistency while the complexity estimator keeps the O(1/sqrt(N)) rate.
- Preferent-compression risk is at most k/N plus O(1/sqrt(N)).
- The logarithmic penalty in earlier bounds is not intrinsic.
- The leading term k/N is necessary.
- The complexity estimator stays consistent when empirical risk does not.
From the original article
1 min read at arxiv.orgCompression schemes provide a general framework for deriving data-dependent risk bounds for learning algorithms. For learning rules based on preferent compression, existing results upper-bound the risk in terms of a sample-dependent complexity $\boldsymbol{k}$, defined as the cardinality of the compression augmented by the training examples on which the reconstructed hypothesis is inappropriate.
By Simone Garatti · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.