Skip to content

Update request builder after pushdown sort into agg buckets #4541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
LantaoJin merged 1 commit into from
Oct 14, 2025
Merged

Update request builder after pushdown sort into agg buckets #4541

LantaoJin merged 1 commit into from
Oct 14, 2025

Conversation

@LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Oct 14, 2025 •
edited
Loading

Description

The PushdownContext is not cloned after creating the action with pushdownSortIntoAggBuckets(). It could make the OpenSearchRequestBuilder out of date, then lead to incorrect results described #4529

Related Issues

Resolves #4529 and #4540

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • New functionality has javadoc added.
  • New functionality has a user manual doc added.
  • New PPL command checklist all confirmed.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff or -s.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin LantaoJin added bug Something isn't working pushdown pushdown related issues labels Oct 14, 2025
@LantaoJin LantaoJin marked this pull request as ready for review October 14, 2025 09:31
@LantaoJin LantaoJin enabled auto-merge (squash) October 14, 2025 09:57
@LantaoJin LantaoJin merged commit fe62472 into opensearch-project:main Oct 14, 2025
58 of 60 checks passed
@opensearch-trigger-bot
Copy link
Contributor

opensearch-trigger-bot bot commented Oct 14, 2025

The backport to 2.19-dev failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4541-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 fe6247277f370bad84f5a70f6445f7dc9774e207
# Push it to GitHub
git push --set-upstream origin backport/backport-4541-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-dev

Then, create a pull request where the base branch is 2.19-dev and the compare/head branch is backport/backport-4541-to-2.19-dev.

LantaoJin added a commit to LantaoJin/search-plugins-sql that referenced this pull request Oct 14, 2025
@LantaoJin
Update request builder after pushdown sort into agg buckets (opensear…
f04370c 
…ch-project#4541)

Signed-off-by: Lantao Jin <[email protected]>
(cherry picked from commit fe62472)
@LantaoJin LantaoJin added the backport-manually Filed a PR to backport manually. label Oct 14, 2025
@LantaoJin LantaoJin mentioned this pull request Oct 14, 2025
Merged
LantaoJin added a commit that referenced this pull request Oct 15, 2025
@LantaoJin
Update request builder after pushdown sort into agg buckets (#4541) (#…
f2e49ee 
…4545)

(cherry picked from commit fe62472)

Signed-off-by: Lantao Jin <[email protected]>
ykmr1224 added a commit to ykmr1224/sql that referenced this pull request Oct 15, 2025
@ykmr1224
Merge main to map-remove:
6acef56 
commit cba8d02
Author: Tomoyuki MORITA <[email protected]>
Date:   Wed Oct 15 13:08:05 2025 -0700

    Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515)

    * Add MAP_APPEND internal function to Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comment

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Rebase and fix IT issue

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 3388dc7
Author: Lantao Jin <[email protected]>
Date:   Thu Oct 16 01:45:29 2025 +0800

    Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569)

    * Use _shard_doc as sort tiebreaker

    Signed-off-by: Lantao Jin <[email protected]>

    * _doc as a part of tie-breaker have better performance

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 5630119
Author: qianheng <[email protected]>
Date:   Wed Oct 15 16:40:41 2025 +0800

    Fix sort push down into agg after project already pushed (opensearch-project#4546)

    * Fix sort push down into agg

    Signed-off-by: Heng Qian <[email protected]>

    * Change some json files to yaml format

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 1e62fba
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 17:20:38 2025 -0700

    Fix JsonExtractAllFunctionIT failure (opensearch-project#4556)

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 02ee33e
Author: Kai Huang <[email protected]>
Date:   Tue Oct 14 14:28:53 2025 -0700

    Add more examples to the `where` command doc (opensearch-project#4457)

    Co-authored-by: Manasvini B S <[email protected]>

commit 0b7e86c
Author: Jialiang Liang <[email protected]>
Date:   Tue Oct 14 10:46:01 2025 -0700

    [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434)

    Co-authored-by: Simeon Widdis <[email protected]>

commit 9c97cfb
Author: Tomoyuki MORITA <[email protected]>
Date:   Tue Oct 14 08:36:43 2025 -0700

    Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489)

    * Add JSON_EXTRACT_ALL internal function for Calcite PPL

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Address comments

    Signed-off-by: Tomoyuki Morita <[email protected]>

    * Minor fix

    Signed-off-by: Tomoyuki Morita <[email protected]>

    ---------

    Signed-off-by: Tomoyuki Morita <[email protected]>

commit 89dbc31
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:24:52 2025 +0800

    Check server status before starting Prometheus (opensearch-project#4537)

    * Check server status before starting Prometheus

    Signed-off-by: Lantao Jin <[email protected]>

    * Change to func call

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix doc

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit fe62472
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 18:10:27 2025 +0800

    Update request builder after pushdown sort into agg buckets (opensearch-project#4541)

    Signed-off-by: Lantao Jin <[email protected]>

commit 42a415f
Author: qianheng <[email protected]>
Date:   Tue Oct 14 17:42:45 2025 +0800

    Including metadata fields type when doing agg/filter script push down (opensearch-project#4522)

    * Including metadata fields type when doing agg/filter script push down

    Signed-off-by: Heng Qian <[email protected]>

    * Fix IT

    Signed-off-by: Heng Qian <[email protected]>

    ---------

    Signed-off-by: Heng Qian <[email protected]>

commit 8de0386
Author: Xinyuan Lu <[email protected]>
Date:   Tue Oct 14 16:41:08 2025 +0800

    Fix percentile bug (opensearch-project#4539)

    * fix percentile bug

    Signed-off-by: xinyual <[email protected]>

    * add IT

    Signed-off-by: xinyual <[email protected]>

    * optimize it

    Signed-off-by: xinyual <[email protected]>

    ---------

    Signed-off-by: xinyual <[email protected]>

commit de2fdc8
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 12:29:03 2025 +0800

    [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534)

    * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited

    Signed-off-by: Lantao Jin <[email protected]>

    * fix doctest

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix conflicts

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

commit 977b7ab
Author: Simeon Widdis <[email protected]>
Date:   Mon Oct 13 20:23:10 2025 -0700

    Update stalled action (opensearch-project#4485)

commit fddbb70
Author: Lantao Jin <[email protected]>
Date:   Tue Oct 14 10:23:12 2025 +0800

    Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501)

    * Add configurable sytem limitations for subsearch and join command

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * typo

    Signed-off-by: Lantao Jin <[email protected]>

    * fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    * remove rollback in doc

    Signed-off-by: Lantao Jin <[email protected]>

    * address comments

    Signed-off-by: Lantao Jin <[email protected]>

    * fix typo

    Signed-off-by: Lantao Jin <[email protected]>

    * Fix IT

    Signed-off-by: Lantao Jin <[email protected]>

    ---------

    Signed-off-by: Lantao Jin <[email protected]>

Signed-off-by: Tomoyuki Morita <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yuancu yuancu yuancu approved these changes

@qianheng-aws qianheng-aws qianheng-aws approved these changes

@ps48 ps48 Awaiting requested review from ps48 ps48 is a code owner

@kavithacm kavithacm Awaiting requested review from kavithacm kavithacm is a code owner

@derek-ho derek-ho Awaiting requested review from derek-ho derek-ho is a code owner

@joshuali925 joshuali925 Awaiting requested review from joshuali925 joshuali925 is a code owner

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric is a code owner

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar vamsimanohar is a code owner

@Swiddis Swiddis Awaiting requested review from Swiddis Swiddis is a code owner

@penghuo penghuo Awaiting requested review from penghuo penghuo is a code owner

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@MaxKsyunz MaxKsyunz Awaiting requested review from MaxKsyunz MaxKsyunz is a code owner

@Yury-Fridlyand Yury-Fridlyand Awaiting requested review from Yury-Fridlyand Yury-Fridlyand is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@forestmvey forestmvey Awaiting requested review from forestmvey forestmvey is a code owner

@acarbonetto acarbonetto Awaiting requested review from acarbonetto acarbonetto is a code owner

@GumpacG GumpacG Awaiting requested review from GumpacG GumpacG is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger is a code owner

@RyanL1997 RyanL1997 Awaiting requested review from RyanL1997 RyanL1997 is a code owner

Assignees

No one assigned

Labels

backport 2.19-dev backport-failed backport-manually Filed a PR to backport manually. bug Something isn't working pushdown pushdown related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[BUG] PPL join on boolean fields returns incomplete results when used with fields command

3 participants

@LantaoJin @yuancu @qianheng-aws