Sparsifying Stochasticity, Not Capacity: Partial Stochasticity via Deep Weight Factorization of Prior Scales

A Bayesian network method sparsifies stochastic parameters via prior-scale factorization while matching fully stochastic accuracy.

The authors learn which Bayesian neural network parameters should be stochastic by applying deep weight factorization to prior scales and fitting a functional prior to a Gaussian process with a maximum mean discrepancy objective. Parameters whose prior scale falls below a cutoff become deterministic, so the regularizer sparsifies stochasticity rather than capacity. They give a linear-time certificate for universal conditional density approximation and show that hybrid sampling-plus-optimization is stochastic approximation of a type-II MAP objective that can retain a nonvanishing tracking error. On UCI benchmarks the method matches a fully stochastic network while keeping about half of its parameters deterministic; random masks with the same budget are worse by up to two orders of magnitude.