feat(search): Hnsw Vector Search Plan Operator & Executor

[Beihao-Zhou]

This PR is part of #2426

Summary

• "Pure" KNN Plan Operator + Executor (i.e. search on HNSW graph directly on disk)

• "Pure" Range Query Plan Operator + Executor

  1. KNN search to get k nearest candidates as a mini-batch initialization, ensuring each one of them is within the expected range
     • If there has been candidates out of the range, then return End;
  2. Expanding results from step 1 by fetching their unvisited neighbours
     • Return results which are within the range and treat them as the starting candidates for the next batch

  Ref: Idea from Pgvector Hnsw Scan Code - bool hnswgettuple(IndexScanDesc scan, ScanDirection dir)

Next

• Hybrid Query with Filter (Potential Solution)

  1. KNN query as usual
  2. Post-process the results by selecting only those results returned by the index that matches the query filter

  Note that since both "Pure" Range Query Executor and "Pure" KNN Search Executor are based on disk, "Pure" Range Query Executor cannot be reused for hybrid query. I plan to add the new hybrid query executor after implementing the ir expression for HNSW (e.g. reuseFilterExecutor for HnswRangeQueryExpr?). Also, I currently don't have a clear idea on how to continue fetching more nearest vectors after some vectors are filtered for hybrid queries. So for hybrid query, I'll make changes in future PRs after the expression layer is more clear to me.

Resources

• Redis Search and Query - Vectors
• HQANN: Efficient and Robust Similarity Search for Hybrid Queries with Structured and Unstructured Constraints
• Filtered − DiskANN: Graph Algorithms for Approximate Nearest Neighbor Search with Filters
• Pgvector Similar Discussion

[Beihao-Zhou]

Hiii @PragmaTwice, once we discussed the potential solutions for vector search query executor on Slack, this PR is attempting to solve it. I'm wondering if the solution above sounds good to you as an initial implementation, and whether you see any room for improvement. Thanks!

[PragmaTwice]

Yeah I think it's good to be an initial implementation. We can gradually refactor it to a more iterative way.

[Beihao-Zhou]

This PR is ready for review! @PragmaTwice
I probably want to add the hybrid executor after implementing the IR expressions (See the Next section in the first conversation block) if this sounds good to you. <3

[PragmaTwice]

This PR is ready for review! @PragmaTwice I probably want to add the hybrid executor after implementing the IR expressions (See the Next section in the first conversation block) if this sounds good to you. <3

Yeah we can then add new nodes in IR and support these new field scan operators in optimizer passes.

e.g. for RediSearch query @year:[2020 2022] @v:[VECTOR_RANGE ...], it's possible to generate a query plan that looks like filter (2020 <= @year and @year <= 2022) from (hnsw-vector-range-scan @v, ...).

As for which query plan is the best, we can leave it to the cost model to determine.

[PragmaTwice]

seems here we can just keep a pointer e.g. const redis::HnswVectorFieldMetadata *field_metadata.

[Beihao-Zhou]

Oh this was because HnswIndex constructor does not support const HnswVectorFieldMetadata* .

HnswIndex(const SearchKey& search_key, HnswVectorFieldMetadata* vector, engine::Storage* storage);

HnswIndex needs to modify HnswVectorFieldMetadata* in other functions, so I did copy here. Do you have ideas on how we can improve it?

[PragmaTwice]

Let me think, if the metadata is modified in a certain vector query, it seems that the global metadata of the vector field (in server->index_mgr->index_map) will not change. Will this cause any problems?

[Beihao-Zhou]

HnswVectorFieldMetadata* is only modified when the node is inserted/deleted in a higher layer than all other nodes, so num_levels in HnswVectorFieldMetadata : IndexFieldMetadata is modified. So the affected field is server->index_mgr->index_map[<index_key>]->fields[<field>]->metadata.

In indexer.cc, we do

auto *metadata = iter->second.metadata.get();
if (auto vector = dynamic_cast<HnswVectorFieldMetadata *>(metadata)) {
    GET_OR_RET(UpdateHnswVectorIndex(key, original, current, search_key, vector));
}

So *metadata changes correspondingly.

But in this PR, since we don't modify metadata, I guess it will not cause actual problems by copying (maybe with consistency issue but this would be fixed later aligning with #2310 , and the expensive copy because of the large size of HnswIndex but I can fix that right after the PR if the solution using static member variable sounds good to you <3).

[PragmaTwice]

I see what you mean.

It's good for now, but we can enhance it later. For example, we could create a const version of HnswIndex that takes a const pointer to HnswVectorFieldMetadata.

[Beihao-Zhou]

That sounds good!
I wanted to overload with a const version, but there are some nested calls also asking for HnswVectorFieldMetadata* as a parameter, so eventually didn't implement it to avoid making this PR look too messy. I'll take a note in tracking issue #2426 and improve it in the future.

[PragmaTwice]

The rest looks good to me. Thank you!

[sonarcloud]

Quality Gate passed

Issues
14 New issues
0 Accepted issues

Measures
0 Security Hotspots
62.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud