Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor add/delete API as an enabler for tiered index #319

Merged
merged 8 commits into from
Dec 29, 2022

Conversation

alonre24
Copy link
Collaborator

@alonre24 alonre24 commented Dec 27, 2022

  • Remove "unmark deleted" API, as it is currently redundant for our async index
  • Let add and delete vector APIs in the index interface to return the number of vectors added/deleted (and take into consideration overwrites).
  • Add an override_allowed flag to addVector interface, so we can enforce that the internal HNSW index within the tiered index will not be able to override existing vectors (rather than creating a delete job in advance).
  • Add indexCapacity() and resize() to index interface and adjust the existing indexes accordingly. Call resize before we add new vectors externally in vec_sim.cpp

@codecov
Copy link

codecov bot commented Dec 27, 2022

Codecov Report

Base: 96.19% // Head: 96.21% // Increases project coverage by +0.01% 🎉

Coverage data is based on head (588616c) compared to base (e299b58).
Patch coverage: 97.14% of modified lines in pull request are covered.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #319      +/-   ##
==========================================
+ Coverage   96.19%   96.21%   +0.01%     
==========================================
  Files          63       63              
  Lines        3526     3536      +10     
==========================================
+ Hits         3392     3402      +10     
  Misses        134      134              
Impacted Files Coverage Δ
src/VecSim/tombstone_interface.h 100.00% <ø> (ø)
src/VecSim/vec_sim_interface.h 100.00% <ø> (ø)
src/VecSim/algorithms/hnsw/hnsw.h 98.94% <92.30%> (-0.01%) ⬇️
src/VecSim/algorithms/brute_force/brute_force.h 100.00% <100.00%> (ø)
.../VecSim/algorithms/brute_force/brute_force_multi.h 96.22% <100.00%> (+0.14%) ⬆️
...VecSim/algorithms/brute_force/brute_force_single.h 98.00% <100.00%> (+0.08%) ⬆️
src/VecSim/algorithms/hnsw/hnsw_multi.h 97.05% <100.00%> (-0.05%) ⬇️
src/VecSim/algorithms/hnsw/hnsw_single.h 98.21% <100.00%> (+0.10%) ⬆️
src/VecSim/algorithms/hnsw/hnsw_tiered.h 41.66% <100.00%> (+5.30%) ⬆️
src/VecSim/vec_sim.cpp 96.66% <100.00%> (+0.04%) ⬆️
... and 6 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@alonre24 alonre24 marked this pull request as ready for review December 28, 2022 17:32
src/VecSim/vec_sim_interface.h Outdated Show resolved Hide resolved
src/VecSim/algorithms/hnsw/hnsw_single.h Outdated Show resolved Hide resolved
src/VecSim/vec_sim_interface.h Outdated Show resolved Hide resolved
tests/unit/test_hnsw.cpp Show resolved Hide resolved
tests/unit/test_hnsw_multi.cpp Outdated Show resolved Hide resolved
Copy link
Collaborator

@DvirDukhan DvirDukhan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please change function comments from override to overwrite

DvirDukhan
DvirDukhan previously approved these changes Dec 29, 2022
@alonre24 alonre24 merged commit 8e6b203 into main Dec 29, 2022
@alonre24 alonre24 deleted the refactor_add_delete_api branch December 29, 2022 15:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants