-
Notifications
You must be signed in to change notification settings - Fork 25.6k
Add notes on indexing to kNN search guide #83188
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -46,15 +46,13 @@ based on a similarity metric, the better its match. | |||||||||||||||||||
| vector function | ||||||||||||||||||||
|
|
||||||||||||||||||||
| In most cases, you'll want to use approximate kNN. Approximate kNN offers lower | ||||||||||||||||||||
| latency and better support for large datasets at the cost of slower indexing and | ||||||||||||||||||||
| reduced accuracy. However, you can configure this method for higher accuracy in | ||||||||||||||||||||
| exchange for slower searches. | ||||||||||||||||||||
| at the cost of slower indexing and imperfect accuracy. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Exact, brute-force kNN guarantees accurate results but doesn't scale well with | ||||||||||||||||||||
| large, unfiltered datasets. With this approach, a `script_score` query must scan | ||||||||||||||||||||
| each matched document to compute the vector function, which can result in slow | ||||||||||||||||||||
| search speeds. However, you can improve latency by using the <<query-dsl,Query | ||||||||||||||||||||
| DSL>> to limit the number of matched documents passed to the function. If you | ||||||||||||||||||||
| large datasets. With this approach, a `script_score` query must scan each | ||||||||||||||||||||
| matching document to compute the vector function, which can result in slow | ||||||||||||||||||||
| search speeds. However, you can improve latency by using a <<query-dsl,query>> | ||||||||||||||||||||
| to limit the number of matching documents passed to the function. If you | ||||||||||||||||||||
| filter your data to a small subset of documents, you can get good search | ||||||||||||||||||||
| performance using this approach. | ||||||||||||||||||||
|
|
||||||||||||||||||||
|
|
@@ -78,8 +76,6 @@ score documents based on similarity between the query and document vector. For a | |||||||||||||||||||
| list of available metrics, see the <<dense-vector-similarity,`similarity`>> | ||||||||||||||||||||
| parameter documentation. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| include::{es-repo-dir}/mapping/types/dense-vector.asciidoc[tag=dense-vector-indexing-speed] | ||||||||||||||||||||
|
|
||||||||||||||||||||
| [source,console] | ||||||||||||||||||||
| ---- | ||||||||||||||||||||
| PUT my-approx-knn-index | ||||||||||||||||||||
|
|
@@ -156,13 +152,29 @@ most similar results from each shard. The search then merges the results from | |||||||||||||||||||
| each shard to return the global top `k` nearest neighbors. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| You can increase `num_candidates` for more accurate results at the cost of | ||||||||||||||||||||
| slower search speeds. A search with a high number of `num_candidates` considers | ||||||||||||||||||||
| more candidates from each shard. This takes more time, but the search has a | ||||||||||||||||||||
| higher probability of finding the true `k` top nearest neighbors. | ||||||||||||||||||||
| slower search speeds. A search with a high value for `num_candidates` | ||||||||||||||||||||
| considers more candidates from each shard. This takes more time, but the | ||||||||||||||||||||
| search has a higher probability of finding the true `k` top nearest neighbors. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| Similarly, you can decrease `num_candidates` for faster searches with | ||||||||||||||||||||
| potentially less accurate results. | ||||||||||||||||||||
|
|
||||||||||||||||||||
| [discrete] | ||||||||||||||||||||
| [[knn-indexing-considerations]] | ||||||||||||||||||||
| ==== Indexing considerations | ||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm very open to suggestions for a better name :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is the reason we put this section in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was wondering this myself and would appreciate @jrodewig's opinion here. I was thinking of this reference as a combined "how to" for kNN search, where we offer guidance about what method to choose, other best practices, etc. I think of the API and field type docs more as specific references that don't give high-level guidance like this. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I think
Yep. That's pretty much my model of these docs. I'm okay with linking from the |
||||||||||||||||||||
|
|
||||||||||||||||||||
| Indexing vectors for approximate kNN search can take substantial time because | ||||||||||||||||||||
| of how expensive it is to build the ANN index structures. You may need to | ||||||||||||||||||||
jtibshirani marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||||||||||||||||
| increase the client request timeout for index and bulk requests. | ||||||||||||||||||||
|
||||||||||||||||||||
| Indexing vectors for approximate kNN search can take substantial time because | |
| of how expensive it is to build the ANN index structures. You may need to | |
| increase the client request timeout for index and bulk requests. | |
| {es} shards are composed of segments, which are internal storage elements in the | |
| index. For approximate kNN search, {es} stores the dense vector values of each | |
| segment as an https://arxiv.org/abs/1603.09320[HNSW graph]. Indexing vectors for | |
| approximate kNN search can take substantial time because of how expensive it is | |
| to build these graphs. You may need to increase the client request timeout for | |
| index and bulk requests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This reorganization works nicely, I'll adopt it.
Uh oh!
There was an error while loading. Please reload this page.