Add documentation for collapse, oversample, truncate_hits processors - #5881
Merged
hdhalter merged 14 commits intoFeb 1, 2024
Conversation
msfroh
requested review from
AMoo-Miki,
Naarcha-AWS,
ananzh,
hdhalter,
kolchfa-aws,
natebower and
vagimeli
as code owners
December 14, 2023 22:52
Signed-off-by: Michael Froh <froh@amazon.com>
msfroh
force-pushed
the
stateful_processor_docs
branch
from
December 14, 2023 22:53
87f7e33 to
f580b30
Compare
hdhalter
added
the
release-notes
kolchfa-aws
approved these changes
Dec 15, 2023
kolchfa-aws
left a comment
kolchfa-aws
left a comment
Collaborator
There was a problem hiding this comment.
Thanks so much, @msfroh! The users will definitely appreciate the clear explanations and a more in-depth example.
|
|
||
| Field | Data type | Description | ||
| :--- | :--- | :--- | ||
| `sample_factor` | Number | The multiplicative factor (>= 1.0) that will be applied to the `size` parameter before processing the search request. Required. |
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
natebower
suggested changes
Dec 18, 2023
natebower
left a comment
natebower
left a comment
Contributor
There was a problem hiding this comment.
@msfroh @kolchfa-aws Please see my comments and changes and let me know if you have any questions. I'd like to read line 18 in the last file before approving, so please tag me when complete. Thanks!
|
|
||
| # Collapse processor | ||
|
|
||
| The `collapse` response processor discards hits that have the same value for some field as a previous document in the result set. |
Contributor
There was a problem hiding this comment.
"that have the same value for a particular field as a previous document in the result set"?
| # Collapse processor | ||
|
|
||
| The `collapse` response processor discards hits that have the same value for some field as a previous document in the result set. | ||
| This is similar to the `collapse` parameter that can be passed in a search request, but the response processor is applied to the |
Contributor
There was a problem hiding this comment.
"This is similar to using the collapse parameter in a search request"?
|
|
||
| ### Collapse without oversample | ||
|
|
||
| In this example, you request the top 3 documents before collapsing on the "color" field. Because the first two documents have the same `color`, the second one is discarded, |
Contributor
There was a problem hiding this comment.
Should "color" be color?
|
|
||
| ### Oversample, collapse, and truncate | ||
|
|
||
| Now, you will use the `oversampling_collapse_pipeline` that requests the top 9 documents (multiplying the size by 3), deduplicates by "color", |
Contributor
There was a problem hiding this comment.
Should "color" be color?
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
reviewed
Dec 18, 2023
Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Collaborator
|
@natebower I addressed all your comments and rewrote the sentence on line 18 as a list of steps. Thanks! |
natebower
reviewed
Dec 18, 2023
natebower
approved these changes
Dec 18, 2023
natebower
left a comment
natebower
left a comment
Contributor
There was a problem hiding this comment.
LGTM with one minor deletion. Thanks!
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com>
kolchfa-aws
added
the
Done but waiting to merge
hdhalter
added
3 - Done
and removed
Done but waiting to merge
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds documentation for the
collapse,oversample, andtrucate_hitssearch pipeline processors.These were added to OpenSearch in opensearch-project/OpenSearch#9405.
Issues Resolved
Fixes #5151
Checklist
For more information on following Developer Certificate of Origin and signing off your commits, please check here.