Paper introduces e-means, a faster mini-batch k-means variant
Original title: e-means: outperforming k-means (in one-tenth the epochs)
Authors report 10-45x GPU speedups and a no-sweep rule for decay rate and batch size.
A paper presents e-means, an exponential moving average procedure for mini-batch k-means that uses deep learning tooling. It derives scaling laws for decay rate and batch size and a formula to set them without a hyperparameter sweep. On embedding datasets, the authors say it reaches a lower loss than Lloyd k-means at 25 epochs in 1.4 to 2.6 epochs, often beating k-means++. They report clustering 10 billion DINO embeddings into 16 million clusters in 2.5 epochs.
Unconfirmed: Speedups and loss comparisons are the authors' reported results, not independently verified.
- e-means uses exponential moving average mini-batch updates.
- Authors give a formula for decay rate and batch size without sweeps.
- Reported GPU wall-clock speedups versus state-of-the-art k-means are 10-45x.
- On DINO10B, 16 million clusters in 2.5 epochs beat Lloyd loss at 25 epochs.
From the original article
1 min read at huggingface.co
Lloyd's k-means algorithm is a long-running standard for clustering, with applications in data analysis, retrieval, quantization and data curation. In this paper, we introduce an exponential moving average optimization procedure for k-means with mini-batch updates, powered by modern deep learning machinery and new insights. We also establish scaling laws on how decay rate and batch size impact final performance.
By Swann Bessa, Pierre-Emmanuel Mazaré, Gergely Szilvasy · huggingface.co
A short excerpt; the article and its images belong to the publisher. Summary, key facts and assessment above are AI-written from the article and can be wrong.