Benign Loss Landscapes Can Coexist with Worst-Case Hardness
Theory paper shows tree tensor networks contain worst-case hard targets yet benign loss landscapes, with difficulty arising from degenerate saddles.
The paper studies tree tensor networks (TTNs), which generalize deep linear networks and Tucker decompositions and embed arbitrary read-once Boolean formulas. It proves that every local minimum that is minimum-norm is global for every realizable target, so bad local minima do not distinguish typical from worst-case problems. Instead, learning difficulty arises from high-order degenerate saddle points caused by rank-deficiency, illustrated via a parity function case study, linking landscape geometry to computational hardness.
- TTNs contain polynomial-size targets unlearnable by gradient descent in polynomial time.
- Every minimum-norm local minimum is global for realizable targets.
- Hardness stems from high-order degenerate saddles caused by rank-deficiency.
- Parity function case study connects landscape geometry to computational hardness.
From the original article
1 min read at arxiv.orgDeep neural networks are expressive enough to contain worst-case targets that can be evaluated in polynomial time but cannot be learned in polynomial time by gradient descent. For practical tasks they nonetheless learn well, raising the question of what non-generic structure of real-world targets enables this.
By Zach Furman · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.