Faiss Training Index takes too long

[mhmd-mst]

Summary

Training Faiss IVFPQ is taking too long and not utilizing my full specs.

Platform

OS: Ubuntu 20.04.3 LTS

Faiss version: 1.6.5

Installed from: YAML file in conda environment

Faiss compilation options:

Running on:

• CPU
• GPU

Interface:

• C++
• Python

Reproduction instructions

d = 128
index = faiss.IndexFlatL2(d)
code_sz = 64
n_centroids = 256
nbits = 8  
max_nitem_train = 1e7
index = faiss.IndexIVFPQ(index, d, n_centroids, code_sz, nbits)
GPU_RESOURCES = faiss.StandardGpuResources()
GPU_OPTIONS = faiss.GpuClonerOptions()
GPU_OPTIONS.useFloat16 = True
index = faiss.index_cpu_to_gpu(GPU_RESOURCES, 0, index, GPU_OPTIONS)
sel_tr_idx = np.random.permutation(len(train_data))
sel_tr_idx = sel_tr_idx[:max_nitem_train]
index.train(train_data[sel_tr_idx, :])

• I have a data saved as memory mapped array but the indexing in last lines change it into numpy array, and i am using 10 M of my data ar random to train where each vector has dimension 128, I have i9-9900K cpu and 32 RAM and RTX 2080Ti, first the training has been running for 20 hours almost, and training is still not done, the data is in VRAM taking around 2GB but the gpu utility spikes to around 30% sometimes but mostly it is under 5%. Is this normal behaviour? And can I do anything to boost the training?
• I am using IVFPQ because using IndexFlatL2 causes out of memory, since my full data is almost 34 GB, when I was using before ubuntu on WSL2 on windows, adding data to the index was consuming both VRAM and RAM, but it is not the case in ubuntu without WSL2 which only uses VRAM in case index is moved to gpu, is there a hybrid approach I can use so that I can stick to FlatL2?
• Regarding the parameters of IndexIVFPQ, is there a specific way to choose them or are they just hyperparameter that I should tune for my data, because with this training time it is gonna take a lot of time.

[mdouze]

For 256 centroids, you should use around 25k training data, see
https://github.com/facebookresearch/faiss/wiki/FAQ#how-many-training-points-do-i-need-for-k-means

[mhmd-mst]

In the suggested link they say if doing training for 6.2 B for 80K centroid as similar to using 20 M to 80 K but they saif before train points should be 1000 x k so shouldnt the train points be 80M instead? Also my vho8ce of centroid is random, my full data has 66.3 M 128-d vectors, how can I choose the number of centroids?
Also anything regarding the other questions I asked previously

[mdouze]

See
https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
https://github.com/facebookresearch/faiss/wiki/FAQ#how-many-training-points-do-i-need-for-k-means