Fix cardinality agg pruning optimization by self collecting #19473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* changes in DenseConjunctionBulkCollector is matching too many documents due to scoreWindow logic * will raise an issue in Lucene to track this * for now collect ourself * this optimization only runs on card agg without sub, and only within the pruning optimization * worst case we can disable this optimization via index setting Signed-off-by: Asim Mahmood <[email protected]>
asimmahmood1
requested review from
a team,
Bukhtawar,
CEHENKLE,
Rishikesh1159,
anasalkouz,
andrross,
ashking94,
cwperks,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
owaiskazi19,
reta,
sachinpkale,
saratvemulapalli,
shwetathareja and
sohami
as code owners
Signed-off-by: Asim Mahmood <[email protected]>
Contributor
Author
|
Tested on nyc_taxis |
rishabhmaurya
approved these changes
Sep 30, 2025
Contributor
|
@asimmahmood1 looks like a good fix. Tbh, i don't mind keeping this logic long term here as dynamic pruning collection logic shouldn't care about any lucene optimizations in future. Should we still create a lucene issue for possible regression if it is applicable in other queries? |
Contributor
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #19473 +/- ##
============================================
- Coverage 73.03% 73.02% -0.01%
+ Complexity 70152 70087 -65
============================================
Files 5683 5683
Lines 321533 321547 +14
Branches 46503 46508 +5
============================================
- Hits 234821 234800 -21
Misses 67790 67790
- Partials 18922 18957 +35 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Author
|
{"run-benchmark-test": "id_1"} |
Contributor
|
{"run-benchmark-test": "id_3"} |
This was referenced Sep 30, 2025
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/4563/ . Final results will be published once the job is completed. |
Contributor
|
The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/4564/ . Final results will be published once the job is completed. |
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/4564/
|
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/165/
|
github-project-automation
bot
moved this from In Progress
to Done
in Performance Roadmap
Collaborator
Benchmark ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/4563/
|
Collaborator
Benchmark Baseline Comparison ResultsBenchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/166/
|
Contributor
Author
|
Benchmark confirms the fix:
|
asimmahmood1
added a commit
to asimmahmood1/OpenSearch
that referenced
this pull request
Sep 30, 2025
…ch-project#19473) Signed-off-by: Asim Mahmood <[email protected]>
Member
+1 Not the first time this optimization gets affected by Lucene change. It's better to collect in this way, directly reuse the related logic from DefaultBulkScorer. |
peteralfonsi
pushed a commit
to peteralfonsi/OpenSearch
that referenced
this pull request
Oct 15, 2025
…ch-project#19473) Signed-off-by: Asim Mahmood <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Related Issues
Fixes [#19367]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.