Ensure topk results for IVF_FLAT_CC#353
Conversation
Signed-off-by: chasingegg <chao.gao@zilliz.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: chasingegg The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| } | ||
| } | ||
| } else if constexpr (std::is_same<IndexType, faiss::IndexIVFFlat>::value) { | ||
| } else if constexpr (std::is_same<IndexType, faiss::IndexIVFFlatCC>::value) { |
There was a problem hiding this comment.
We're not using IndexIVFFlat, just IndexIVFFlatCC all the time, right?
There was a problem hiding this comment.
Actually I dont know why we need a seperate code branch for IndexIVFFlat, it is the same with IVFPQ, IVFSQ, etc
There was a problem hiding this comment.
But we do need a new code branch for IndexIVFFlatCC, so I replace it.
| struct SearchParametersIVF : SearchParameters { | ||
| size_t nprobe = 1; ///< number of probes at query time | ||
| size_t max_codes = 0; ///< max nb of codes to visit to do a query | ||
| bool ensure_topk_full = false; ///< indicate whether we make sure topk result is full |
There was a problem hiding this comment.
I'd add more comments here. It did not become clear what this parameter does before I read the code :)
There was a problem hiding this comment.
Sure, I will add more comments in the next pr.
| idx_t* labels, | ||
| size_t k) const; | ||
| size_t k, | ||
| size_t& scan_cnt) const; |
There was a problem hiding this comment.
Please add comments to the description of the function.
|
/lgtm |
Yeah, I've opened an issue at faiss repo, it is actually a break change to current faiss implementation on 'scan_count' meaning, and I somehow prefer the meaning in knowhere :) |
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Signed-off-by: chasingegg <chao.gao@zilliz.com>
* Remove omp (#276) Signed-off-by: Yudong Cai <yudong.cai@zilliz.com> * [cherry-pick]Pip3 install in requirements.txt order (#303) Signed-off-by: cqy123456 <qianya.cheng@zilliz.com> * Cherry-pick #294 #298 from main (#300) Signed-off-by: Yudong Cai <yudong.cai@zilliz.com> * Deprecate Invalid config checking (#304) Signed-off-by: Li Liu <li.liu@zilliz.com> * Fix scann range search (#316) Signed-off-by: chasingegg <chao.gao@zilliz.com> * Upgrade conan to 1.61.0 (#182) (#347) Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com> Co-authored-by: Enwei Jiao <enwei.jiao@zilliz.com> * raft hasrawdata return false Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com> * switch knowhere-test branch (#384) Signed-off-by: cqy123456 <qianya.cheng@zilliz.com> * Ensure topk results for IVF_FLAT_CC (#353) (#383) Signed-off-by: chasingegg <chao.gao@zilliz.com> * make sure we rethrow exceptions in async tasks: make sure we do not crash due to uncaught exceptions when we called folly::Future::wait but not trying to get the values; use folly::collect to simplify code (#382) Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> * [2.2 fix] wrap IVF index train/build calls in lambdas passed to knowhere thread pool, so OMP threads spawned will have low nice values (#379) Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> * fix diskann async cache generation (#377) Signed-off-by: xianliang <xianliang.li@zilliz.com> * fix:miss wait thread tasks finish in diskann. (#380) Signed-off-by: cqy123456 <qianya.cheng@zilliz.com> * Shuffle ids for building hnsw index (#322) (#381) Signed-off-by: chasingegg <chao.gao@zilliz.com> * sync knowherer 2.2.4 Signed-off-by: xianliang <xianliang.li@zilliz.com> --------- Signed-off-by: Yudong Cai <yudong.cai@zilliz.com> Signed-off-by: cqy123456 <qianya.cheng@zilliz.com> Signed-off-by: Li Liu <li.liu@zilliz.com> Signed-off-by: chasingegg <chao.gao@zilliz.com> Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com> Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com> Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com> Signed-off-by: xianliang <xianliang.li@zilliz.com> Co-authored-by: Cai Yudong <yudong.cai@zilliz.com> Co-authored-by: cqy123456 <39671710+cqy123456@users.noreply.github.com> Co-authored-by: liliu-z <105927039+liliu-z@users.noreply.github.com> Co-authored-by: Gao <chao.gao@zilliz.com> Co-authored-by: Enwei Jiao <enwei.jiao@zilliz.com> Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com> Co-authored-by: Buqian Zheng <zhengbuqian@gmail.com>
3 participants
#354
/kind improvement