KL Before and After Mixing: A Round-Complexity Separation in Parallel Sampling
Theory paper separates two KL measures for parallel sampling, with very different round costs.
The authors compare the schedule-averaged KL divergence of a parallel sampler with the KL divergence of its mixture output law. For uniform even-length binary strings with half the bits set, the expected round count is on the order of (log n) squared under the first criterion but only about e to the minus epsilon times log log n under the second. The separation extends to weighted fixed-count targets and a full-support Beta-Bernoulli model, where the sharp output-law cost is about half of e to the minus epsilon times log log n.
- Compares schedule-averaged KL with KL of the mixture output.
- Balanced binary strings need Theta of (log n) squared rounds.
- The output-law criterion needs only double-logarithmic rounds.
- The separation extends to weighted and Beta-Bernoulli targets.
From the original article
1 min read at arxiv.orgA parallel sampler can choose at random which coordinates to generate together. For a target $p$ and the output law $q_U$ of a schedule $U$ drawn afresh for each output, we compare the schedule-averaged divergence $\mathbb{E}_U D_{\mathrm{KL}}(p\|q_U)$ with the output-law divergence $D_{\mathrm{KL}}(p\|\mathbb{E}_U q_U)$, in nats.
By Tianyi Wu · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.