Refactor k-NN filter search by kolchfa-aws · Pull Request #3613 · opensearch-project/documentation-website

kolchfa-aws · 2023-03-29T17:27:15Z

Fixes #3221

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws · 2023-03-29T17:27:51Z

@martin-gaievski Could you please review this PR for technical accuracy?

martin-gaievski · 2023-03-29T18:29:22Z

Please add following example of scoring script query so out doc will have sample for all three ways of filtering:

{
    "size": 3,
    "query": {
        "script_score": {
            "query": {
                "bool": {
                    "filter": {
                        "bool": {
                            "must": [
                                {
                                    "range": {
                                        "rating": {
                                            "gte": 8,
                                            "lte": 10
                                        }
                                    }
                                },
                                {
                                    "term": {
                                        "parking": "true"
                                    }
                                }
                            ]
                        }
                    }
                }
            },
            "script": {
                "source": "knn_score",
                "lang": "knn",
                "params": {
                    "field": "location",
                    "query_value": [
                        5.0,
                        4.0
                    ],
                    "space_type": "l2"
                }
            }
        }
    }
}

martin-gaievski · 2023-03-29T18:31:41Z

_search-plugins/knn/filter-search-knn.md

+- [Boolean filter](#boolean-filter-with-ann-search): This approach runs an [Approximate nearest neighbor (ANN)]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn) search, and then applies the filter to the results. Because it uses post-filtering, a query with a Boolean filter may return significantly less than `k` results for a restrictive filter.

-The OpenSearch k-NN plugin version 2.2 introduced support for the Lucene engine in order to process k-NN searches. The Lucene engine provides a search that is based on the HNSW algorithm in order to represent a multi-layered graph. The OpenSearch k-NN plugin version 2.4 can incorporate filters for searches based on Lucene 9.4.
+- [Lucene k-NN filter](#using-a-lucene-k-nn-filter): This approach ensures that `k` results are returned because filtering is applied during the k-NN search as opposed to after the k-NN search, like in post-filtering. The drawback is that you can only use this method with the Hierarchical Navigable Small World (HNSW) algorithm implemented by a Lucene search engine.


Please mention that lucene filtering is available from 2.4.

martin-gaievski · 2023-03-29T18:33:43Z

_search-plugins/knn/filter-search-knn.md

+## Filtered search optimization

-Lucene also provides the capability to operate its `KnnVectorQuery` across a subset of documents. To learn more about this capability, see the [Apache Lucene Documentation](https://issues.apache.org/jira/browse/LUCENE-10382).
+Overall, Lucene k-NN filters are more efficient compared to other filtering methods, both in terms of performance and relevancy of search results. 


This statement isn't correct for all use cases, and later we provide table where each of 3 filtering types paired with the dataset and filter combination.

martin-gaievski · 2023-03-29T18:34:54Z

_search-plugins/knn/filter-search-knn.md

+Overall, Lucene k-NN filters are more efficient compared to other filtering methods, both in terms of performance and relevancy of search results. 

-To learn more about all available k-NN search approaches, including approximate k-NN, exact k-NN with script score, and pre-filtering with painless extensions, see [k-NN]({{site.url}}{{site.baseurl}}/search-plugins/knn/index/).
+Depending on your dataset and use case, you might be more interested in maximizing recall or minimizing latency. The following table provides guidance on various k-NN search configurations and the filtering methods to use in order to optimize for better recall or lower latency. The first three columns of the table provide several example k-NN search configurations. A search configuration consists of:


I think we need to state "higher recall" if it follows by "lower latency".

martin-gaievski · 2023-03-29T18:36:23Z

_search-plugins/knn/filter-search-knn.md

-
-Number of Vectors | Filter Restrictive Percentage | k | Recall | Latency
-- | -- | -- | -- | --
+Number of documents in an index | Percentage of documents the filter returns | k | Filtering method to use for best recall | Filtering method to use for lowest latency


same as above, not sure we need to say "best recall", probably "higher recall" is more accurate

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws · 2023-03-29T19:10:22Z

@martin-gaievski I have added the scoring script example (changed the query_value to use a double array because integers raised an exception) and addressed the comments. Please re-review when you get a chance and approve the PR when everything looks good. Thanks!

martin-gaievski

Looks good to me, thank you

cwillum

Looks great.

cwillum · 2023-03-31T23:11:06Z

_search-plugins/knn/filter-search-knn.md

-
-Number of Vectors | Filter Restrictive Percentage | k | Recall | Latency
-- | -- | -- | -- | --
+Number of documents in an index | Percentage of documents the filter returns | k | Filtering method to use for higher recall | Filtering method to use for lower latency


Jeff has got everyone going on this program: add pipes before and after all table elements (in case we ever switch to a host that can only interpret a markdown table with this format. Not sure if you think that's worthwhile.

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

natebower

@kolchfa-aws Please see my comments and changes and let me know if you have any questions. Thanks!

_search-plugins/knn/filter-search-knn.md

natebower · 2023-04-05T12:43:22Z

_search-plugins/knn/filter-search-knn.md

-#### Example request
-
-The following request returns hotels that provide parking. This request illustrates multiple alternative mechanisms to obtain the parking filter criteria. It uses a regular expression for the value `true`, a term query for the key-value pair `"parking":"true"`, a wildcard for the characters that spell "true", and the `must_not` clause to eliminate hotels with "parking" set to `false`:
+The following request illustrates alternative mechanisms to search for hotels with parking using a `term`, `wildcard`, and `regexp` query clauses in the `should` clause. Additionally, it uses the `must_not` clause to eliminate hotels with `parking` set to `false`:


Suggested change

The following request illustrates alternative mechanisms to search for hotels with parking using a `term`, `wildcard`, and `regexp` query clauses in the `should` clause. Additionally, it uses the `must_not` clause to eliminate hotels with `parking` set to `false`:

The following request illustrates how to search for hotels with parking using `term`, `wildcard`, and `regexp` query clauses in the `should` clause. Additionally, it uses the `must_not` clause to eliminate hotels with `parking` set to `false`:

Co-authored-by: Nathan Bower <nbower@amazon.com>

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

* Refactor k-NN filter search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented doc review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> * One more editorial review comment Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit ade705e) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Refactor k-NN filter search * Implemented tech review comments * Implemented doc review feedback * Apply suggestions from code review * One more editorial review comment --------- (cherry picked from commit ade705e) Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

* Refactor k-NN filter search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented doc review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> * One more editorial review comment Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

This reverts commit 481883f.

* Refactor k-NN filter search Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented tech review comments Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Implemented doc review feedback Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> * One more editorial review comment Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> --------- Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

Refactor k-NN filter search

da43c3a

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws requested a review from a team as a code owner March 29, 2023 17:27

kolchfa-aws self-assigned this Mar 29, 2023

martin-gaievski reviewed Mar 29, 2023

View reviewed changes

Implemented tech review comments

06a5f11

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

martin-gaievski approved these changes Mar 31, 2023

View reviewed changes

cwillum approved these changes Mar 31, 2023

View reviewed changes

vagimeli approved these changes Apr 4, 2023

View reviewed changes

Implemented doc review feedback

5c4a035

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

natebower reviewed Apr 5, 2023

View reviewed changes

kolchfa-aws and others added 2 commits April 5, 2023 10:16

Apply suggestions from code review

85469a0

Co-authored-by: Nathan Bower <nbower@amazon.com>

One more editorial review comment

28cef9d

Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>

kolchfa-aws merged commit ade705e into main Apr 5, 2023

kolchfa-aws added backport 2.4 PR: Backport label for 2.4 backport 2.5 PR: Backport label for 2.5 backport 2.6 PR: Backport label for 2.6 labels Apr 5, 2023

This was referenced Apr 5, 2023

[Backport 2.6] Refactor k-NN filter search #3678

Merged

[Backport 2.4] Refactor k-NN filter search #3679

Merged

opensearch-trigger-bot bot mentioned this pull request Apr 5, 2023

[Backport 2.5] Refactor k-NN filter search #3680

Merged

vagimeli added a commit that referenced this pull request Apr 25, 2023

Revert "Refactor k-NN filter search (#3613)"

5cda322

This reverts commit 481883f.

Naarcha-AWS deleted the knn-fix branch March 28, 2024 23:20

	The following request illustrates alternative mechanisms to search for hotels with parking using a `term`, `wildcard`, and `regexp` query clauses in the `should` clause. Additionally, it uses the `must_not` clause to eliminate hotels with `parking` set to `false`:
	The following request illustrates how to search for hotels with parking using `term`, `wildcard`, and `regexp` query clauses in the `should` clause. Additionally, it uses the `must_not` clause to eliminate hotels with `parking` set to `false`:

Conversation

kolchfa-aws commented Mar 29, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

kolchfa-aws commented Mar 29, 2023

Uh oh!

martin-gaievski commented Mar 29, 2023

Uh oh!

martin-gaievski Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

martin-gaievski Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

martin-gaievski Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

martin-gaievski Mar 29, 2023

Choose a reason for hiding this comment

Uh oh!

kolchfa-aws commented Mar 29, 2023

Uh oh!

martin-gaievski left a comment

Choose a reason for hiding this comment

Uh oh!

cwillum left a comment

Choose a reason for hiding this comment

Uh oh!

cwillum Mar 31, 2023

Choose a reason for hiding this comment

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

natebower Apr 5, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kolchfa-aws commented Mar 29, 2023 •

edited

Loading