Merging index files failing on Windows 10

[FlimFlamm]

Summary

Attempting to merge a set of index files results in the following error:

RuntimeError: Error in struct faiss::InvertedListsIOHook *__cdecl faiss::InvertedListsIOHook::lookup(int) at D:\a\faiss-wheels\faiss-wheels\faiss\faiss\invlists\InvertedListsIOHook.cpp:63: read_InvertedLists: could not load ArrayInvertedLists as 646f6c69 ("ilod") 

Fulsome error at bottom of post.

Description:
I am following the following guide from a third party repository: https://github.com/rom1504/clip-retrieval/blob/main/docs/laion5B_h14_back.md in an attempt to run a local image-captioning pipeline. My issue occurs when I try to combine the 80 index files into one larger index, which uses faiss to do so through the following python function:

def index_combiner(input_folder, output_folder):
    """combine the index files into a single index file"""
    index_dir = Path(input_folder)
    block_fnames = sorted([str(a) for a in index_dir.glob("*") if "index" in str(a)])
    empty_index = faiss.read_index(block_fnames[0], faiss.IO_FLAG_MMAP)
    empty_index.ntotal = 0

    if not os.path.exists(output_folder):
        os.makedirs(output_folder)

    merge_ondisk(empty_index, block_fnames, output_folder + "/merged_index.ivfdata")

    faiss.write_index(empty_index, output_folder + "/populated.index")

The problem is that it breaks on this line: empty_index = faiss.read_index(block_fnames[0], faiss.IO_FLAG_MMAP) and throws the error shown above (and at bottom of post).

My best current guess as to the basic cause of the issue (other than corrupt index files possibly), is that since the overall index is 700 GB in total, well outside of my memory limitations, my windows OS might be the cause #2106 . I can't seem to find solid details about what exactly causes this, how I can get around it, or what my alternatives might be.

Platform

OS: Windows 10

Faiss version: faiss-cpu=1.7.4

Installed from: third party package: (pip install huggingface_hub clip-retrieval)

Faiss compilation options: uncertain (set by third party package I reckon)

Running on:

• CPU
• Python

Reproduction instructions

Reproducing this properly involves downloading over 700 gigabytes, so i don't recommend trying, but simply follow this guide on a windows 10 system and it will lead to the following error

(clipmancer2) C:\Users\chris\Documents\cadmancer\local_clip>clip-retrieval index_combiner --input_folder "index-parts" --output_folder "combined-indices"
Traceback (most recent call last):
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\chris\miniconda3\envs\clipmancer2\Scripts\clip-retrieval.exe\__main__.py", line 7, in <module>
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\clip_retrieval\cli.py", line 18, in main
    fire.Fire(
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\fire\core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\fire\core.py", line 466, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\clip_retrieval\clip_back_prepro\index_combiner.py", line 14, in index_combiner
    empty_index = faiss.read_index(block_fnames[0], faiss.IO_FLAG_MMAP)
  File "C:\Users\chris\miniconda3\envs\clipmancer2\lib\site-packages\faiss\swigfaiss.py", line 9795, in read_index
    return _swigfaiss.read_index(*args)
RuntimeError: Error in struct faiss::InvertedListsIOHook *__cdecl faiss::InvertedListsIOHook::lookup(int) at D:\a\faiss-wheels\faiss-wheels\faiss\faiss\invlists\InvertedListsIOHook.cpp:63: read_InvertedLists: could not load ArrayInvertedLists as 646f6c69 ("ilod")

Please don't hesitate to ask for any additional information that might be

[mdouze]

OnDisk indexes is not supported on windows because the OS does not support the mmap function.

[FlimFlamm]

Thanks for the confirmation on this; as a result I've decided I might as well take the Ubuntu plunge...

But out of curiosity, do you know if there are alternative methods to combining this kind of index file?

[mdouze]

If they fit in RAM, then you can just load the indexes and call merge_from.