How to increase the number of nodes sampled in training KMeans?

[namespace-Pt]

I want to perform kmeans over 8M nodes into 10k clusters. Faiss defaults to sample 2560000 points but I'd like to use the entire dataset. What's the solution?

[kunal-kotian]

faiss.Kmeans has a property max_points_per_centroid which is set to 256 by default. With k clusters, this means only k * 256 datapoints can be used for fitting kmeans. In your case, this turns out to be 2560000 datapoints, which get subsampled from your full dataset. To use all 8M samples for fitting Kmeans, just pass max_points_per_centroid=800 to the faiss.Kmeans() constructor.

For reference, see this link: https://github.com/facebookresearch/faiss/wiki/FAQ#can-i-ignore-warning-clustering-xxx-points-to-yyy-centroids