Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersion
Corrected two-level softmax samplers remove cluster-size and dispersion bias with little overhead.
Two-level softmax sampling first draws a cluster and then an item, enabling sublinear-time sampling from large softmax distributions. The authors show that standard two-level sampling systematically misweights clusters by ignoring size imbalance and intra-cluster similarity dispersion. Size-Corrected 2LS and Size- and Dispersion-Corrected 2LS correct those biases and provide provably better softmax approximations with negligible overhead. Experiments on five large-scale datasets support using the corrected methods instead of standard two-level sampling.
- Standard two-level softmax ignores cluster size and intra-cluster dispersion.
- S-2LS and SD-2LS correct that bias with negligible added cost.
- Experiments on five large datasets show improved sampling fidelity.
From the original article
1 min read at arxiv.orgSampling from a softmax distribution is a fundamental operation in machine learning, but its linear complexity in the number of items makes exact sampling impractical at scale. Two-level softmax (2LS) sampling is a popular alternative enabling sublinear-time sampling. Assuming items are partitioned into clusters, 2LS first samples a cluster and then an item within it.
By Walid Bendada · arxiv.org
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.