Merged
Conversation
Summary: Pull Request resolved: #3170 Logging info, adding to heap and wait_in and out times. Reviewed By: algoriddle Differential Revision: D52034667 fbshipit-source-id: 8ab864c5c43d534d094c6e81bb810c74e20c9ac2
Summary: This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/mlomeli1/faiss/blob/oivf/demos/offline_ivf/README.md) for details about using this framework. This PR includes the following unit tests, which can be run with the unittest library as so: ```` ~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back ```` In test_offline_ivf: ```` test_consistency_check test_train_index test_index_shard_equal_file_sizes test_index_shard_unequal_file_sizes test_search test_evaluate_without_margin test_evaluate_without_margin_OPQ test_evaluate_with_margin test_split_batch_size_bigger_than_file_sizes test_split_batch_size_smaller_than_file_sizes test_split_files_with_corrupted_input_file ```` In test_iterate_input: ```` test_iterate_input_file_larger_than_batch test_get_vs_iterate test_iterate_back ```` Pull Request resolved: #3175 Reviewed By: algoriddle Differential Revision: D52218447 Pulled By: mlomeli1 fbshipit-source-id: 78b12457c79b02eb2c9ae993560f2e295798e7e5
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state. Reviewed By: bunnypak Differential Revision: D52355753 fbshipit-source-id: 54c8b47224c1dafc4e2464fe4e528c78ecabf0b8
Summary: `-Wunused-function` has identified an unused function. This diff removes it. In many cases these functions have existed for years in an unused state. Reviewed By: bunnypak Differential Revision: D52355736 fbshipit-source-id: e315cf420053472218bbf2fdd647ab073c6f6cd6
Summary: Revert so that the test data can be replaced with synthetic Reviewed By: mdouze Differential Revision: D52388604 fbshipit-source-id: c0037635a4e66c54d42400294d13d9a80610b845
…amming_distance/generic-inl.h Summary: ## Context Due to a recent change (D52393687) to increase compiler warnings arcoss faiss/PACKAGE aggregator [build](https://www.internalfb.com/chronos/job/gp/9007207455707765) is failing. The annotation [[fallthrough]]; needs to be used in every [nested]case statement where fallthrough can occur. ## COMPILER ERROR ``` [2023-12-22T22:06:48.478-08:00] Action failed: fbcode//faiss:faiss (ovr_config//platform/linux:x86_64-fbcode-platform010-clang15-jemalloc-master#a916d57931e3a679) (cxx_compile impl/LocalSearchQuantizer.cpp) [CONTEXT] [2023-12-22T22:06:48.478-08:00] Remote command returned non-zero exit code 1 [CONTEXT] [2023-12-22T22:06:48.478-08:00] Remote action, reproduce with: `frecli cas download-action d02e5a31e82ab2a6a2ce8b84dff4d4bce3997ca49783013f860d16dd566edaaf:145` buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:278:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough] [CONTEXT] case 7: [CONTEXT] ^ buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:281:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough] [CONTEXT] case 6: [CONTEXT] ^ buck-out/v2/gen/fbcode/a916d57931e3a679/faiss/__faiss__/buck-headers/faiss/utils/hamming_distance/generic-inl.h:284:21: error: unannotated fall-through between switch labels [-Werror,-Wimplicit-fallthrough] [CONTEXT] case 5: ``` ## WHAT I DID +1 annotation **[[fallthrough]];** at every case statement that required it. ## WHAT I DIDNT DO - I did NOT **EXPORTED TO OSS FAISS** - mdouze please do this if it is required. I do not know your policy. Prior to landing i saw the message but was unsure. As well as not having github account. Differential Revision: D52396801 fbshipit-source-id: a563068af00135f7fd24de6633007a1957fc6e31
…amming_distance/neon-inl.h Summary: Error: https://www.internalfb.com/sandcastle/workflow/1661828262501424314 Due to a recent change (D52393687) to increase compiler warnings arcoss faiss/PACKAGE aggregator build is failing. The annotation [[fallthrough]]; needs to be used in every [nested]case statement where fallthrough can occur. WHAT I DIDNT DO I did NOT EXPORTED TO OSS FAISS mdouze please do this if it is required. I do not know your policy. Prior to landing i saw the message but was unsure. As well as not having github account. Reviewed By: r-barnes Differential Revision: D52420730 fbshipit-source-id: 92ec8382226ea35bdacb668b3c329bd8b41a67fc
…nl.h Differential Revision: D52436745 fbshipit-source-id: 7e4c55a8255b2f25b3394146a24d1af698b3b2e9
…tatsD.cpp
Summary:
`-Wunused-exception-parameter` has identified an unused exception parameter. This diff removes it.
This:
```
try {
...
} catch (exception& e) {
// no use of e
}
```
should instead be written as
```
} catch (exception&) {
```
If the code compiles, this is safe to land.
Reviewed By: dmm-fb
Differential Revision: D51977749
fbshipit-source-id: aab6f8bcb8a0d9273b7894b8e85615d4a31cac86
Summary: - IVF benchmarks: `bench_fw_ivf.py bench_fw_ivf.py bigann /checkpoint/gsz/bench_fw/ivf` - Codec benchmarks: `bench_fw_codecs.py contriever /checkpoint/gsz/bench_fw/codecs` and `bench_fw_codecs.py deep1b /checkpoint/gsz/bench_fw/codecs` - A range codec evaluation: `bench_fw_range.py ssnpp /checkpoint/gsz/bench_fw/range` - Visualize with `bench_fw_notebook.ipynb` - Support for running on a cluster Pull Request resolved: #3189 Reviewed By: mdouze Differential Revision: D52544642 Pulled By: algoriddle fbshipit-source-id: 21dcdfd076aef6d36467c908e6be78ef851b0e98
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: dmm-fb Differential Revision: D52582819 fbshipit-source-id: 34c4d5eca31bcc2df5872c9eb3a286335c8ae131
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: algoriddle Differential Revision: D52582890 fbshipit-source-id: ef7e44805fb212c6415030b98f65c00b2fd4a9b7
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: algoriddle Differential Revision: D52582797 fbshipit-source-id: 4ade95834fee68a4ac86b4d03a00df0760941f4e
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: algoriddle Differential Revision: D52582823 fbshipit-source-id: 575c286c0445e2388038aac10327d6c1bd63d6df
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: algoriddle Differential Revision: D52582910 fbshipit-source-id: 92ee06e819f15c411a3fee75a6c9e17fba96a7f3
Summary: Pull Request resolved: #3197 Reviewed By: mlomeli1 Differential Revision: D52689379 Pulled By: algoriddle fbshipit-source-id: 54e27c6d310d6da14777ae10ae62f46e7076cacf
…#3190) Summary: Pull Request resolved: #3190 This diff adds more result handlers in order to expose them externally. This enables range search for HSNW and Fast Scan, and nprobe parameter support for FastScan. Reviewed By: pemazare Differential Revision: D52547384 fbshipit-source-id: 271da5ffea6411df3d8e50641abade18bd7b774b
Summary: ### Description We have SIMD support for x86 architecture using AVX2 optimization. But, we don't have a similar optimization for ARM architecture in Scalar Quantizer for the Quantization Type `QT_FP16`. This PR adds SIMD support for ARM using NEON optimization. ### Issues resolved #3014 Pull Request resolved: #3166 Reviewed By: algoriddle, pemazare Differential Revision: D52510486 Pulled By: mdouze fbshipit-source-id: 2bb360475a0d9e0816c8e05b44d7da7f2e6b28c5
Summary: Diff summary courtesy of Metamate: This diff bumps the version of libraft to fix nightly issues in the faiss-gpu-raft package. The code changes include updating the libraft version in the meta.yaml file to 24.02. This change is necessary to ensure that the package is compatible with the latest version of libraft. Reviewed By: mlomeli1 Differential Revision: D52736355 fbshipit-source-id: 7ac2427210c06eb2d1c3ab4a3d37d48494fb039e
Summary: LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance. This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52734573 fbshipit-source-id: edf315a4e2e7ddfaf8b879b7f4a3fc3a6c2e607f
Summary: LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance. This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52734585 fbshipit-source-id: be5da3411d2965f6cd2516376598576049c03f09
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582916 fbshipit-source-id: 367ea85b569063efa50c82702c3ba5efa2fb92f9
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582928 fbshipit-source-id: 57bd901e87cbb8ddfd423c8ae6baefe1048c206f
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582813 fbshipit-source-id: 02a742df17614c546e10446bc530835e876942f0
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: palmje Differential Revision: D52582927 fbshipit-source-id: be03fb1354898d0637d287f285ea99d096c39008
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582870 fbshipit-source-id: 4c81218e109ff97052036645ab33bc9b17802e91
Summary: This PR introduces the offline IVF (OIVF) framework which contains some tooling to run search using IVFPQ indexes (plus OPQ pretransforms) for large batches of queries using [big_batch_search](https://github.com/mlomeli1/faiss/blob/main/contrib/big_batch_search.py) and GPU faiss. See the [README](https://github.com/facebookresearch/faiss/blob/36226f5fe8412162863d7d5d4cab97809f4d211d/demos/offline_ivf/README.md) for details about using this framework. This PR includes the following unit tests, which can be run with the unittest library as so: ```` ~/faiss/demos/offline_ivf$ python3 -m unittest tests/test_iterate_input.py -k test_iterate_back ```` In test_offline_ivf: ```` test_consistency_check test_train_index test_index_shard_equal_file_sizes test_index_shard_unequal_file_sizes test_search test_evaluate_without_margin test_evaluate_without_margin_OPQ ```` In test_iterate_input: ```` test_iterate_input_file_larger_than_batch test_get_vs_iterate test_iterate_back ```` Pull Request resolved: #3202 Reviewed By: algoriddle Differential Revision: D52734222 Pulled By: mlomeli1 fbshipit-source-id: 61fd0084277c1b14bdae1189db8ae43340611e16
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582837 fbshipit-source-id: 275d20f940102030989f9bcfb4884891457b149c
Summary: Our upcoming compiler upgrade will require us not to have shadowed variables. Such variables have a _high_ bug rate and reduce readability, so we would like to avoid them even if the compiler was not forcing us to do so. This codemod attempts to fix an instance of a shadowed variable. Please review with care: if it's failed the result will be a silent bug. **What's a shadowed variable?** Shadowed variables are variables in an inner scope with the same name as another variable in an outer scope. Having the same name for both variables might be semantically correct, but it can make the code confusing to read! It can also hide subtle bugs. This diff fixes such an issue by renaming the variable. - If you approve of this diff, please use the "Accept & Ship" button :-) Reviewed By: meyering Differential Revision: D52582914 fbshipit-source-id: edb3983635de4343c4a0b20096e81852cfa058c6
Summary: reference the new Faiss paper in the docs. Also added Gergely as a co-author of Faiss. Some minor updates. Reviewed By: mlomeli1 Differential Revision: D52829321 fbshipit-source-id: 0f1845beace6cd88f809ba50b87a8d446475d30b
Summary: Pull Request resolved: #3473 Previous diff (D57602154) fixed the CircleCI version and the PR build version of GHA but not the nightly one. Reviewed By: junjieqi Differential Revision: D57680576 fbshipit-source-id: 39f49c20df824c915f536b1ed3ffc35db2907988
Summary: Pull Request resolved: #3474 This commit focus on the cpp version of PQfastscan tutorial with index refinement by defining the k factor. Reviewed By: junjieqi Differential Revision: D57680905 fbshipit-source-id: 980c2990172f24ec9a4f870685e706195883408f
Summary: mdouze Please let me know if any additional unit tests are needed Pull Request resolved: #3444 Reviewed By: algoriddle Differential Revision: D57665641 Pulled By: mdouze fbshipit-source-id: 9bec91306a1c31ea4f1f1d726c9d60ac6415fdfc
Summary: Pull Request resolved: #3475 Migration to GitHub Reviewed By: algoriddle Differential Revision: D57707064 fbshipit-source-id: 17f0a97028007f3664faa5b6b2c269f50bcdf39e
Summary: Pull Request resolved: #3476 The long tail will still be the RAFT build but we can save on cost if the build has errors by incurring ~10m penalty added by waiting for the basic cmake build to complete. Both GPU and RAFT builds will start together so this will take less time overall to complete. Reviewed By: algoriddle Differential Revision: D57707298 fbshipit-source-id: 3589842e9bda9ebca9b25e089e6177fe96b6a0f5
Summary: Pull Request resolved: #3477 AVX-512 must remain on CircleCI until GitHub provides runners with AVX-512 support (ETA: Q1 2025). Reviewed By: algoriddle Differential Revision: D57707621 fbshipit-source-id: e8a0885f8363cf8f20854cccca3ec0adc946362b
Summary: Pull Request resolved: #3450 Reviewed By: mdouze Differential Revision: D57708412 Pulled By: junjieqi fbshipit-source-id: 9540b7e60d8b2b39e0ca92423d2a305fab2a17e6
Summary: Small Raft related modification to StandardGpuResources: if the stream for a particular device is modified by a user, delete the Raft handle for that device. On any subsequent call to `getRaftHandle(device)`, a new raft handle with the updated stream will be created. Closes #3424 Pull Request resolved: #3435 Reviewed By: ramilbakhshyiev Differential Revision: D57640976 Pulled By: junjieqi fbshipit-source-id: 41e2898a39250b7e52e920b71e819fc21ca9fc85
…cy measurement (#3480) Summary: Pull Request resolved: #3480 This tutorial summarize the methods to construct different indexs for PQFastScan refinement. It shows how the choice can impact on accuracy. Reviewed By: junjieqi Differential Revision: D57799598 fbshipit-source-id: a75c52c60a5217366f3361676da8f03f0c4a9feb
Summary:
simple math:
| **input n** | **input nshards** | shard_size | idx | i0 | ni |
| -- |-- |-- |-- |-- |-- |
| 19 | 6 | 4 | 5 | 20 | **-1** |
| 1000 | 37 | 28 | 36 | 1008 | -8 |
| 1000 | 64 | 16 | 63 | 1008 | -8 |
root cause:
integer cause precision loss, `idx * shard_size` overflows, because `(n + nshards - 1) / nshards` is roundup
my solution:
each shard takes at least `base_shard_size = n / nshards`, then `remain = n % nshards`, we know `0 <= remain < nshards`, next, assign those remain vectors to first `remain` shards, i.e. first `remain` shards take one more vector each.
```c++
auto i0 = idx * base_shard_size;
if (i0 < remain) {
// if current idx is one of the first `remain` shards
i0 += idx;
} else {
i0 += remain;
}
```
simplify above code: `i0 = idx * base_shard_size + std::min(size_t(idx), n % nshards);`
Pull Request resolved: #3374
Reviewed By: fxdawnn
Differential Revision: D57867910
Pulled By: junjieqi
fbshipit-source-id: 7e72ea5cd197af4f3446fb7a3fd34ad08901dbb2
Summary: Pull Request resolved: #3422 Found vec_io failing when running some benchmarking. There is no such field named big_endian in sys. So, reverting it to original field byteorder Reviewed By: algoriddle Differential Revision: D56718607 fbshipit-source-id: 553f1d2d6bc967581142a92282e534f3f164e8f9
Summary: Pull Request resolved: #3423 Adding small fixes to run experiments from fbcode. 1. Added buck target 2. Full import path of faiss bench_fw modules 3. new dataset path to run tests locally as we can't use an existing directory ./data in fbcode. Reviewed By: algoriddle, junjieqi Differential Revision: D57235092 fbshipit-source-id: f78a23199e619b640a19ca37f8b52ff0abdd8298
Summary: Pull Request resolved: #3487 Sometimes it is not useful to serialize the storage index along with a HNSW index. This diff adds a flag that supports skipping the storage of the index. Searchign and adding to the index is not possible until a storage index is added back in. Reviewed By: junjieqi Differential Revision: D57911060 fbshipit-source-id: 5a4ceee4a8f53f6f746df59af3942b813a99c14f
Summary: Pull Request resolved: #3494 This tasks focus on the refine index construction tutorial with different index refinement on fp16/sq8 quantization. The python version was added a while ago. Reviewed By: junjieqi Differential Revision: D58161983 fbshipit-source-id: 1c598fe612b5dee3952c5f7398e6802e117f141d
Summary: Adding build folder to gitignore, so that they don't show up in the commit tree while building from source Pull Request resolved: #3492 Reviewed By: junjieqi Differential Revision: D58171359 Pulled By: asadoughi fbshipit-source-id: b0efed348769328a3bdbcc13098dcb84cadb6c4f
Summary: Pull Request resolved: #3502 include probably added by vscode Reviewed By: mengdilin Differential Revision: D58411537 fbshipit-source-id: 3035f690d26decc937fb492c54ffa2f974ee2db8
Summary: Add support for `use_raft` in the torch version of `knn_gpu`. The numpy version already has this support, see https://github.com/facebookresearch/faiss/blob/main/faiss/python/gpu_wrappers.py#L59 Pull Request resolved: #3509 Reviewed By: mlomeli1, junjieqi Differential Revision: D58489851 Pulled By: algoriddle fbshipit-source-id: cfad722fefd4809b135b765d0d43587cfd782d0e
Summary: Pull Request resolved: #3512 Pull Request resolved: #3510 GitHub hosted runners some with the build-essentials package pre-installed, self-hosted runners on AWS do not have this package. This made it all steps other than the `all targets` one fall back to the system executables which unintentially worked on GitHub hosted runners but not on the self-hosted ones. This diff fixes it by pulling the line that adds conda bin to path early in the cmake build action. Reviewed By: asadoughi Differential Revision: D58513853 fbshipit-source-id: 23e95459e0031c96bd142515db07d1b700d713cf
Summary: Pull Request resolved: #3515 Fix typo `test_io_no_storage` Reviewed By: kuarora, asadoughi Differential Revision: D58540190 fbshipit-source-id: b8b9cacd7ea6005c0edb94014de74188450318c1
) Summary: Pull Request resolved: #3516 This diff seeks to simplify the steps that install conda packages and environment configuration into a single step at the start of the cmake build action. Reviewed By: mnorris11 Differential Revision: D58560454 fbshipit-source-id: ee2c6b36865809f31eb335cfb3c2fffdccaa318d
Summary: Pull Request resolved: #3517 MSVC doesn't support unsigned int for OMP Reviewed By: kuarora, junjieqi, ramilbakhshyiev Differential Revision: D58591594 fbshipit-source-id: ac7d6b37a82f9543be3e0fe418f0f6b439751475
Summary: Pull Request resolved: #3519 Fixing the conda conflicts because of `_openmp_mutex` build versions. This change pins that version for RAFT conda package builds. Reviewed By: algoriddle Differential Revision: D58646659 fbshipit-source-id: 4c1eaa9f08bd354da016b9399a36698007a497d8
Summary: Pull Request resolved: #3522 Quick fix to unblock nightly Reviewed By: mlomeli1 Differential Revision: D58694193 fbshipit-source-id: ea323991cc2e2b958fc11ab614dcd6e09d4c072c
Summary: Pull Request resolved: #3524 Searches with the metric abs(dot(query, database)) This makes it possible to search vectors that are closest to a hyperplane * adds support for alternative metrics in faiss.knn in python * checks that it works with HNSW * simplifies the extra distances interface by removing the template on Reviewed By: asadoughi Differential Revision: D58695971 fbshipit-source-id: 2a0ff49c7f7ac2c005d85f141cc5de148081c9c4
tarang-jain
pushed a commit
that referenced
this pull request
Jul 5, 2024
…ookresearch#3527) Summary: Pull Request resolved: facebookresearch#3527 **Context** Design Doc: [Faiss Benchmarking](https://docs.google.com/document/d/1c7zziITa4RD6jZsbG9_yOgyRjWdyueldSPH6QdZzL98/edit) **In this diff** 1. Be able to reference codec and index from blobstore (bucket & path) outside the experiment 2. To support #1, naming is moved to descriptors. 3. Build index can be written as well. 4. You can run benchmark with train and then refer it in index built and then refer index built in knn search. Index serialization is optional. Although not yet exposed through index descriptor. 5. Benchmark can support index with different datasets sizes 6. Working with varying dataset now support multiple ground truth. There may be small fixes before we could use this. 7. Added targets for bench_fw_range, ivf, codecs and optimize. **Analysis of ivf result**: D58823037 Reviewed By: algoriddle Differential Revision: D57236543 fbshipit-source-id: ad03b28bae937a35f8c20f12e0a5b0a27c34ff3b
tarang-jain
pushed a commit
that referenced
this pull request
Jul 5, 2024
…ookresearch#3527) Summary: Pull Request resolved: facebookresearch#3527 **Context** Design Doc: [Faiss Benchmarking](https://docs.google.com/document/d/1c7zziITa4RD6jZsbG9_yOgyRjWdyueldSPH6QdZzL98/edit) **In this diff** 1. Be able to reference codec and index from blobstore (bucket & path) outside the experiment 2. To support #1, naming is moved to descriptors. 3. Build index can be written as well. 4. You can run benchmark with train and then refer it in index built and then refer index built in knn search. Index serialization is optional. Although not yet exposed through index descriptor. 5. Benchmark can support index with different datasets sizes 6. Working with varying dataset now support multiple ground truth. There may be small fixes before we could use this. 7. Added targets for bench_fw_range, ivf, codecs and optimize. **Analysis of ivf result**: D58823037 Reviewed By: algoriddle Differential Revision: D57236543 fbshipit-source-id: ad03b28bae937a35f8c20f12e0a5b0a27c34ff3b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fetch upstream main