Add a new interface method SecondaryIndex::NewIterator to enable querying the index #13257
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D67684777 |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 28, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value; however, other semantics are also possible. For vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (The new API will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 28, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value; however, other semantics are also possible. For vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (The new API will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
force-pushed
the
export-D67684777
branch
from
1c9c4ae to
17565da
Compare
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 28, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value; however, other semantics are also possible. For vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (The new API will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
force-pushed
the
export-D67684777
branch
from
17565da to
8021135
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67684777 |
1 similar comment
|
This pull request was exported from Phabricator. Differential Revision: D67684777 |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 28, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value; however, other semantics are also possible. For vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (The new API will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 28, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value; however, other semantics are also possible. For vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (The new API will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 30, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
force-pushed
the
export-D67684777
branch
from
8021135 to
5427aa7
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67684777 |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 30, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 30, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 30, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 31, 2024
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Dec 31, 2024
Summary: The patch is the read-side counterpart of facebook#13197 . It adds support for K-nearest-neighbor vector similarity searches to `FaissIVFIndex`. There are two main pieces to this: 1) `KNNIterator` is an `Iterator` implementation that is returned by `FaissIVFIndex` upon a call to `NewIterator`. `KNNIterator` treats its `Seek` target as a vector embedding and passes it to FAISS along with the number of neighbors requested `k` as well as the number of probes to use (i.e. the number of inverted lists to check). Applications can then use `Next` (and `Prev`) to iterate over the the vectors in the result set. `KNNIterator` exposes the primary keys associated with the result vectors (see below how this is done), while `value` and `columns` are empty. The iterator also supports a property `rocksdb.faiss.ivf.index.distance` that can be used to retrieve the distance/similarity metric for the current result vector. 2) `IteratorAdapter` takes a RocksDB secondary index iterator (see facebook#13257) and adapts it to the interface required by FAISS (`faiss::InvertedListsIterator`), enabling FAISS to read the inverted lists stored in RocksDB. Since FAISS only supports numerical vector ids of type `faiss::idx_t`, `IteratorAdapter` uses `KNNIterator` to assign ephemeral (per-query) ids to the inverted list items read during iteration, which are later mapped back to the original primary keys by `KNNIterator`. Differential Revision: D67684898
jaykorean
reviewed
Jan 2, 2025
| // For vector indices, the search target might be a vector, and the iterator | ||
| // might return similar vectors from the index. | ||
| virtual std::unique_ptr<Iterator> NewIterator( | ||
| const ReadOptions& read_options, Iterator* underlying_it) const = 0; |
There was a problem hiding this comment.
Just wondering if we are going to guarantee that underlying iterator stays available. Or is it going to be the API caller's responsibility to keep the underlying iter alive as long as this secondary index iterator is being used.
There was a problem hiding this comment.
Right now it's the latter but it might actually make sense for the secondary index iterator to take ownership (i.e. for NewIterator to take a unique_ptr<Iterator>&&). Let me change that.
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
force-pushed
the
export-D67684777
branch
from
5427aa7 to
06430d0
Compare
|
This pull request was exported from Phabricator. Differential Revision: D67684777 |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 2, 2025
…ying the index (facebook#13257) Summary: The patch adds a new API `NewIterator` to `SecondaryIndex`, which should return an iterator that can be used by applications to query the index. This method takes a `ReadOptions` structure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.) Querying the index can be performed by calling the returned iterator's `Seek` API with a search target, and then using `Next` (and potentially `Prev`) to iterate through the matching index entries. `SeekToFirst`, `SeekToLast`, and `SeekForPrev` are not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries. The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class `SecondaryIndexIterator`.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented for `FaissIVFIndex` in a subsequent patch.) Differential Revision: D67684777
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 2, 2025
Summary: The patch is the read-side counterpart of facebook#13197 . It adds support for K-nearest-neighbor vector similarity searches to `FaissIVFIndex`. There are two main pieces to this: 1) `KNNIterator` is an `Iterator` implementation that is returned by `FaissIVFIndex` upon a call to `NewIterator`. `KNNIterator` treats its `Seek` target as a vector embedding and passes it to FAISS along with the number of neighbors requested `k` as well as the number of probes to use (i.e. the number of inverted lists to check). Applications can then use `Next` (and `Prev`) to iterate over the the vectors in the result set. `KNNIterator` exposes the primary keys associated with the result vectors (see below how this is done), while `value` and `columns` are empty. The iterator also supports a property `rocksdb.faiss.ivf.index.distance` that can be used to retrieve the distance/similarity metric for the current result vector. 2) `IteratorAdapter` takes a RocksDB secondary index iterator (see facebook#13257) and adapts it to the interface required by FAISS (`faiss::InvertedListsIterator`), enabling FAISS to read the inverted lists stored in RocksDB. Since FAISS only supports numerical vector ids of type `faiss::idx_t`, `IteratorAdapter` uses `KNNIterator` to assign ephemeral (per-query) ids to the inverted list items read during iteration, which are later mapped back to the original primary keys by `KNNIterator`. Differential Revision: D67684898
jaykorean
reviewed
Jan 2, 2025
| // building block for more complex iterators. | ||
| class SecondaryIndexIterator : public Iterator { | ||
| public: | ||
| SecondaryIndexIterator(const SecondaryIndex* index, |
There was a problem hiding this comment.
I see that ReadOptions are not being passed here yet. I assume that will come in the later PRs?
There was a problem hiding this comment.
Right, this implementation doesn't use any read options; however, for example, the index iterator for vector similarity search will
jaykorean
reviewed
Jan 2, 2025
| @@ -96,6 +99,32 @@ class SecondaryIndex { | |||
| const Slice& primary_column_value, const Slice& previous_column_value, | |||
| std::optional<std::variant<Slice, std::string>>* secondary_value) | |||
| const = 0; | |||
|
|
|||
| // Create an iterator that can be used by applications to query the index. | |||
There was a problem hiding this comment.
I think we'd want to add EXPERIMENTAL tag
There was a problem hiding this comment.
The entire secondary indexing functionality (including this class) is currently marked as being "under construction" but hopefully will graduate to "experimental" soon ;)
|
Thanks for the review! |
|
This pull request has been merged in 3579d32. |
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 2, 2025
Summary: The patch is the read-side counterpart of facebook#13197 . It adds support for K-nearest-neighbor vector similarity searches to `FaissIVFIndex`. There are two main pieces to this: 1) `KNNIterator` is an `Iterator` implementation that is returned by `FaissIVFIndex` upon a call to `NewIterator`. `KNNIterator` treats its `Seek` target as a vector embedding and passes it to FAISS along with the number of neighbors requested `k` as well as the number of probes to use (i.e. the number of inverted lists to check). Applications can then use `Next` (and `Prev`) to iterate over the the vectors in the result set. `KNNIterator` exposes the primary keys associated with the result vectors (see below how this is done), while `value` and `columns` are empty. The iterator also supports a property `rocksdb.faiss.ivf.index.distance` that can be used to retrieve the distance/similarity metric for the current result vector. 2) `IteratorAdapter` takes a RocksDB secondary index iterator (see facebook#13257) and adapts it to the interface required by FAISS (`faiss::InvertedListsIterator`), enabling FAISS to read the inverted lists stored in RocksDB. Since FAISS only supports numerical vector ids of type `faiss::idx_t`, `IteratorAdapter` uses `KNNIterator` to assign ephemeral (per-query) ids to the inverted list items read during iteration, which are later mapped back to the original primary keys by `KNNIterator`. Differential Revision: D67684898
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 2, 2025
Summary: The patch is the read-side counterpart of facebook#13197 . It adds support for K-nearest-neighbor vector similarity searches to `FaissIVFIndex`. There are two main pieces to this: 1) `KNNIterator` is an `Iterator` implementation that is returned by `FaissIVFIndex` upon a call to `NewIterator`. `KNNIterator` treats its `Seek` target as a vector embedding and passes it to FAISS along with the number of neighbors requested `k` as well as the number of probes to use (i.e. the number of inverted lists to check). Applications can then use `Next` (and `Prev`) to iterate over the the vectors in the result set. `KNNIterator` exposes the primary keys associated with the result vectors (see below how this is done), while `value` and `columns` are empty. The iterator also supports a property `rocksdb.faiss.ivf.index.distance` that can be used to retrieve the distance/similarity metric for the current result vector. 2) `IteratorAdapter` takes a RocksDB secondary index iterator (see facebook#13257) and adapts it to the interface required by FAISS (`faiss::InvertedListsIterator`), enabling FAISS to read the inverted lists stored in RocksDB. Since FAISS only supports numerical vector ids of type `faiss::idx_t`, `IteratorAdapter` uses `KNNIterator` to assign ephemeral (per-query) ids to the inverted list items read during iteration, which are later mapped back to the original primary keys by `KNNIterator`. Differential Revision: D67684898
ltamasi
added a commit
to ltamasi/rocksdb
that referenced
this pull request
Jan 4, 2025
Summary: The patch is the read-side counterpart of facebook#13197 . It adds support for K-nearest-neighbor vector similarity searches to `FaissIVFIndex`. There are two main pieces to this: 1) `KNNIterator` is an `Iterator` implementation that is returned by `FaissIVFIndex` upon a call to `NewIterator`. `KNNIterator` treats its `Seek` target as a vector embedding and passes it to FAISS along with the number of neighbors requested `k` as well as the number of probes to use (i.e. the number of inverted lists to check). Applications can then use `Next` (and `Prev`) to iterate over the the vectors in the result set. `KNNIterator` exposes the primary keys associated with the result vectors (see below how this is done), while `value` and `columns` are empty. The iterator also supports a property `rocksdb.faiss.ivf.index.distance` that can be used to retrieve the distance/similarity metric for the current result vector. 2) `IteratorAdapter` takes a RocksDB secondary index iterator (see facebook#13257) and adapts it to the interface required by FAISS (`faiss::InvertedListsIterator`), enabling FAISS to read the inverted lists stored in RocksDB. Since FAISS only supports numerical vector ids of type `faiss::idx_t`, `IteratorAdapter` uses `KNNIterator` to assign ephemeral (per-query) ids to the inverted list items read during iteration, which are later mapped back to the original primary keys by `KNNIterator`. Differential Revision: D67684898
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The patch adds a new API
NewIteratortoSecondaryIndex, which should return an iterator that can be used by applications to query the index. This method takes aReadOptionsstructure, which can be used by applications to provide (implementation-specific) query parameters to the index, and an underlying iterator, which should be an iterator over the index's secondary column family, and is expected to be leveraged by the returned iterator to read the actual secondary index entries. (Providing the underlying iterator this way enables querying the index as of a specific point in time for example.)Querying the index can be performed by calling the returned iterator's
SeekAPI with a search target, and then usingNext(and potentiallyPrev) to iterate through the matching index entries.SeekToFirst,SeekToLast, andSeekForPrevare not expected to be supported by the iterator. The iterator should expose primary keys, that is, the secondary key prefix should be stripped from the index entries.The exact semantics of the returned iterator depend on the index and are implementation-specific. For simple indices, the search target might be a primary column value, and the iterator might return all primary keys that have the given column value. (This behavior can be achieved using the new class
SecondaryIndexIterator.) However, other semantics are also possible: for vector indices, the search target might be a vector, and the iterator might return similar vectors from the index. (This will be implemented forFaissIVFIndexin a subsequent patch.)Differential Revision: D67684777