Including metadata fields type when doing agg/filter script push down #4522
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Heng Qian <[email protected]>
qianheng-aws
requested review from
GumpacG,
LantaoJin,
MaxKsyunz,
RyanL1997,
Swiddis,
YANG-DB,
Yury-Fridlyand,
acarbonetto,
anirudha,
dai-chen,
derek-ho,
forestmvey,
joshuali925,
kavithacm,
mengweieric,
noCharger,
penghuo,
ps48,
seankao-az,
vamsimanohar,
ykmr1224 and
yuancu
as code owners
Signed-off-by: Heng Qian <[email protected]>
LantaoJin
reviewed
Oct 13, 2025
opensearch/src/main/java/org/opensearch/sql/opensearch/storage/OpenSearchIndex.java
Show resolved
Hide resolved
| Content-Type: 'application/json' | ||
| ppl: | ||
| body: | ||
| query: source=test | eval id_int = cast(_id as int) | where id_int > 1 |
Member
There was a problem hiding this comment.
no issue for ppl source=test | eval id_int = cast(_id as int) | sort id_int?
Collaborator
Author
There was a problem hiding this comment.
The sort operator on derived field isn't able to be pushed down for now.
Member
There was a problem hiding this comment.
then what if source=test | eval id_int = cast(_id as int) | stats count() by span(id_int, 100)
LantaoJin
approved these changes
Oct 13, 2025
yuancu
approved these changes
Oct 14, 2025
Contributor
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/sql/backport-2.19-dev 2.19-dev
# Navigate to the new working tree
pushd ../.worktrees/sql/backport-2.19-dev
# Create a new branch
git switch --create backport/backport-4522-to-2.19-dev
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 42a415fc3f3f439b535fb7ffd688b3d81f0244ff
# Push it to GitHub
git push --set-upstream origin backport/backport-4522-to-2.19-dev
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/sql/backport-2.19-devThen, create a pull request where the |
qianheng-aws
added a commit
to qianheng-aws/sql
that referenced
this pull request
Oct 15, 2025
…opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> (cherry picked from commit 42a415f) Signed-off-by: Heng Qian <[email protected]>
LantaoJin
added
backport-manually
yuancu
pushed a commit
that referenced
this pull request
Oct 15, 2025
…er script push down (#4522) (#4566) * Including metadata fields type when doing agg/filter script push down (#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> (cherry picked from commit 42a415f) Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]>
ykmr1224
added a commit
to ykmr1224/sql
that referenced
this pull request
Oct 15, 2025
commit cba8d02 Author: Tomoyuki MORITA <[email protected]> Date: Wed Oct 15 13:08:05 2025 -0700 Add MAP_APPEND internal function to Calcite PPL (opensearch-project#4515) * Add MAP_APPEND internal function to Calcite PPL Signed-off-by: Tomoyuki Morita <[email protected]> * Minor fix Signed-off-by: Tomoyuki Morita <[email protected]> * Address comment Signed-off-by: Tomoyuki Morita <[email protected]> * Rebase and fix IT issue Signed-off-by: Tomoyuki Morita <[email protected]> --------- Signed-off-by: Tomoyuki Morita <[email protected]> commit 3388dc7 Author: Lantao Jin <[email protected]> Date: Thu Oct 16 01:45:29 2025 +0800 Use `_doc` + `_shard_doc` as sort tiebreaker to get better performance (opensearch-project#4569) * Use _shard_doc as sort tiebreaker Signed-off-by: Lantao Jin <[email protected]> * _doc as a part of tie-breaker have better performance Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> commit 5630119 Author: qianheng <[email protected]> Date: Wed Oct 15 16:40:41 2025 +0800 Fix sort push down into agg after project already pushed (opensearch-project#4546) * Fix sort push down into agg Signed-off-by: Heng Qian <[email protected]> * Change some json files to yaml format Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> commit 1e62fba Author: Tomoyuki MORITA <[email protected]> Date: Tue Oct 14 17:20:38 2025 -0700 Fix JsonExtractAllFunctionIT failure (opensearch-project#4556) Signed-off-by: Tomoyuki Morita <[email protected]> commit 02ee33e Author: Kai Huang <[email protected]> Date: Tue Oct 14 14:28:53 2025 -0700 Add more examples to the `where` command doc (opensearch-project#4457) Co-authored-by: Manasvini B S <[email protected]> commit 0b7e86c Author: Jialiang Liang <[email protected]> Date: Tue Oct 14 10:46:01 2025 -0700 [Enhancement] Error handling for illegal character usage in java regex named capture group (opensearch-project#4434) Co-authored-by: Simeon Widdis <[email protected]> commit 9c97cfb Author: Tomoyuki MORITA <[email protected]> Date: Tue Oct 14 08:36:43 2025 -0700 Add JSON_EXTRACT_ALL internal function for Calcite PPL (opensearch-project#4489) * Add JSON_EXTRACT_ALL internal function for Calcite PPL Signed-off-by: Tomoyuki Morita <[email protected]> * Address comments Signed-off-by: Tomoyuki Morita <[email protected]> * Minor fix Signed-off-by: Tomoyuki Morita <[email protected]> --------- Signed-off-by: Tomoyuki Morita <[email protected]> commit 89dbc31 Author: Lantao Jin <[email protected]> Date: Tue Oct 14 18:24:52 2025 +0800 Check server status before starting Prometheus (opensearch-project#4537) * Check server status before starting Prometheus Signed-off-by: Lantao Jin <[email protected]> * Change to func call Signed-off-by: Lantao Jin <[email protected]> * Fix doc Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> commit fe62472 Author: Lantao Jin <[email protected]> Date: Tue Oct 14 18:10:27 2025 +0800 Update request builder after pushdown sort into agg buckets (opensearch-project#4541) Signed-off-by: Lantao Jin <[email protected]> commit 42a415f Author: qianheng <[email protected]> Date: Tue Oct 14 17:42:45 2025 +0800 Including metadata fields type when doing agg/filter script push down (opensearch-project#4522) * Including metadata fields type when doing agg/filter script push down Signed-off-by: Heng Qian <[email protected]> * Fix IT Signed-off-by: Heng Qian <[email protected]> --------- Signed-off-by: Heng Qian <[email protected]> commit 8de0386 Author: Xinyuan Lu <[email protected]> Date: Tue Oct 14 16:41:08 2025 +0800 Fix percentile bug (opensearch-project#4539) * fix percentile bug Signed-off-by: xinyual <[email protected]> * add IT Signed-off-by: xinyual <[email protected]> * optimize it Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]> commit de2fdc8 Author: Lantao Jin <[email protected]> Date: Tue Oct 14 12:29:03 2025 +0800 [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited (opensearch-project#4534) * [FollowUp] Set 0 and negative value of subsearch.maxout as unlimited Signed-off-by: Lantao Jin <[email protected]> * fix doctest Signed-off-by: Lantao Jin <[email protected]> * Fix conflicts Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> commit 977b7ab Author: Simeon Widdis <[email protected]> Date: Mon Oct 13 20:23:10 2025 -0700 Update stalled action (opensearch-project#4485) commit fddbb70 Author: Lantao Jin <[email protected]> Date: Tue Oct 14 10:23:12 2025 +0800 Add configurable sytem limitations for `subsearch` and `join` command (opensearch-project#4501) * Add configurable sytem limitations for subsearch and join command Signed-off-by: Lantao Jin <[email protected]> * Fix IT Signed-off-by: Lantao Jin <[email protected]> * typo Signed-off-by: Lantao Jin <[email protected]> * fix IT Signed-off-by: Lantao Jin <[email protected]> * remove rollback in doc Signed-off-by: Lantao Jin <[email protected]> * address comments Signed-off-by: Lantao Jin <[email protected]> * fix typo Signed-off-by: Lantao Jin <[email protected]> * Fix IT Signed-off-by: Lantao Jin <[email protected]> --------- Signed-off-by: Lantao Jin <[email protected]> Signed-off-by: Tomoyuki Morita <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Including metadata fields type when doing agg/filter script push down, as to avoid NPE when there is metadata fields in the script.
Related Issues
Resolves #4513
Check List
--signoffor-s.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.