Researchers have introduced e-means, a new optimization method for the k-means clustering problem that they say reaches a lower error than the standard Lloyd algorithm in a small fraction of the training time. In tests across a wide range of datasets, e-means reached a lower test loss than Lloyd does after 25 epochs in as few as 1.2 to 3 epochs.

How much faster is e-means than Lloyd's algorithm?

The speed gains hold up when measured end to end. The authors report a 10 to 45 times reduction in wall-clock time on GPU against Flash-KMeans, which they describe as the state-of-the-art Lloyd implementation. On CPU, they measured an 8 to 15 times reduction against Faiss. Clustering methods usually force a choice between speed and quality, the researchers note, but e-means delivers both better speed and a better loss.

How does e-means differ from Lloyd's algorithm?

K-means works by finding a set of centroids that minimize the average squared distance between each data point and its nearest centroid. Lloyd’s algorithm solves that problem by reassigning every point on each pass through the full dataset. E-means instead works on mini-batches and uses exponential moving averages to update each cluster’s running sum and count. The team says this approach, which is typically used for shifting data distributions, is also highly efficient on static datasets when combined with other changes.

Those changes include a new initialization method, a revised strategy for reviving clusters that go empty, a cosine-scheduled decay rate and a finalization stage that lowers training loss at the end of a run for a small added runtime cost. The method also sets its own batch size and step size using two scaling laws fit to a problem’s number of points, number of clusters and number of epochs. That means practitioners do not need to set any extra hyperparameters compared with Lloyd’s algorithm.

Does e-means avoid poor local solutions more often than Lloyd?

The authors also tested whether e-means avoids poor local solutions more often than Lloyd. Across 200 random starts on toy datasets, Lloyd got stuck in a suboptimal configuration on 26 percent of starts in a case with two blobs of very different sizes, while e-means did so on 3.5 percent. In a nine-blob grid with heavier corners, Lloyd stayed stuck more than 1 percent above the best error on 43 percent of starts, compared with 4 percent for e-means. On a line of five blobs, Lloyd stayed stuck on 46 percent of starts, while e-means got stuck on none. The researchers attribute the difference to noise. Mini-batch updates are more likely to explore the optimization landscape, they write, while Lloyd’s steadily decreasing loss makes it more likely to settle into the first minimum it finds.

Dead clusters, centroids that end up with no assigned points, are a persistent problem for mini-batch methods, and they matter more as codebooks grow. In one test on a million vectors with 16,384 clusters, leaving revival off produced 4,313 dead clusters, or 26.3 percent of the codebook. A common fix, which replaces dead centroids with random samples from the incoming batch, cut that to 2,343. E-means instead splits a large, existing cluster in two, chosen with probability proportional to its size, and left only 47 dead clusters on the same dataset.

The team validated the method on 24 embedding and synthetic datasets and configurations. On the 11 datasets of the VIBE embedding benchmark, which covers text, code, image and multimodal embeddings, e-means needed between 1.6 and 2.4 epochs to match the test error Lloyd reached in 25 epochs.

The code is available in the Faiss repository under faiss/contrib/e_means, with both PyTorch and NumPy versions. Developers call the Emeans class with the number of clusters and epochs, and the default is five epochs. The paper, arXiv:2611.09501, lists Swann Bessa, Pierre-Emmanuel Mazaré, Gergely Szilvasy, Hugo Touvron, David Picard, Matthijs Douze and Hervé Jégou as authors.

Questo articolo è stato scritto con l'assistenza dell'IA.
News Factory APP - notizie agentiche per potenziare il tuo SEO e AEO.