Support `appendpipe`command in PPL #4602

xinyual · 2025-10-20T07:50:38Z

Description

Add appendpipe command.
Appendpipe appends the result of the subpipeline to the search results. Unlike a subsearch, the subpipeline is not run first. The subpipeline is run when the search reaches the appendpipe command.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
New PPL command checklist all confirmed.
API changes companion pull request created.
Commits are signed per the DCO using --signoff or -s.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: xinyual <[email protected]>

yuancu

Review in progress

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java

Signed-off-by: xinyual <[email protected]>

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendPipeCommandIT.java

Signed-off-by: xinyual <[email protected]>

yuancu

LGTM

xinyual · 2025-11-05T06:54:40Z

@Swiddis could you please approve again?

qianheng-aws · 2025-11-07T02:42:06Z

core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java

+    while (childNode.getChild() != null
+        && !childNode.getChild().isEmpty()
+        && !(childNode.getChild().getFirst() instanceof Values)) {
+      childNode = (UnresolvedPlan) childNode.getChild().getFirst();


Will there be more than 1 children in the subquery? Maybe adding assertion and throw exception to avoid this case if we don't support.

Signed-off-by: xinyual <[email protected]>

* add demos Signed-off-by: xinyual <[email protected]> * add missing column Signed-off-by: xinyual <[email protected]> * add appendpipe poc Signed-off-by: xinyual <[email protected]> * slighty change syntax Signed-off-by: xinyual <[email protected]> * add unresolved plan Signed-off-by: xinyual <[email protected]> * add IT Signed-off-by: xinyual <[email protected]> * add tests Signed-off-by: xinyual <[email protected]> * remove useless ut Signed-off-by: xinyual <[email protected]> * fix conflict Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * remove useless chaneg Signed-off-by: xinyual <[email protected]> * add explain IT Signed-off-by: xinyual <[email protected]> * fix IT Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * add doc Signed-off-by: xinyual <[email protected]> * optimize doc Signed-off-by: xinyual <[email protected]> * add UT Signed-off-by: xinyual <[email protected]> * fix IT due to performance change Signed-off-by: xinyual <[email protected]> * add multiply children check Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]> (cherry picked from commit 3a3c8c8) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Support `appendpipe`command in PPL (#4602) * add demos Signed-off-by: xinyual <[email protected]> * add missing column Signed-off-by: xinyual <[email protected]> * add appendpipe poc Signed-off-by: xinyual <[email protected]> * slighty change syntax Signed-off-by: xinyual <[email protected]> * add unresolved plan Signed-off-by: xinyual <[email protected]> * add IT Signed-off-by: xinyual <[email protected]> * add tests Signed-off-by: xinyual <[email protected]> * remove useless ut Signed-off-by: xinyual <[email protected]> * fix conflict Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * remove useless chaneg Signed-off-by: xinyual <[email protected]> * add explain IT Signed-off-by: xinyual <[email protected]> * fix IT Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * add doc Signed-off-by: xinyual <[email protected]> * optimize doc Signed-off-by: xinyual <[email protected]> * add UT Signed-off-by: xinyual <[email protected]> * fix IT due to performance change Signed-off-by: xinyual <[email protected]> * add multiply children check Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]> * avoid to use get first Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>

* add demos Signed-off-by: xinyual <[email protected]> * add missing column Signed-off-by: xinyual <[email protected]> * add appendpipe poc Signed-off-by: xinyual <[email protected]> * slighty change syntax Signed-off-by: xinyual <[email protected]> * add unresolved plan Signed-off-by: xinyual <[email protected]> * add IT Signed-off-by: xinyual <[email protected]> * add tests Signed-off-by: xinyual <[email protected]> * remove useless ut Signed-off-by: xinyual <[email protected]> * fix conflict Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * remove useless code Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * remove useless chaneg Signed-off-by: xinyual <[email protected]> * add explain IT Signed-off-by: xinyual <[email protected]> * fix IT Signed-off-by: xinyual <[email protected]> * apply spotless Signed-off-by: xinyual <[email protected]> * add doc Signed-off-by: xinyual <[email protected]> * optimize doc Signed-off-by: xinyual <[email protected]> * add UT Signed-off-by: xinyual <[email protected]> * fix IT due to performance change Signed-off-by: xinyual <[email protected]> * add multiply children check Signed-off-by: xinyual <[email protected]> --------- Signed-off-by: xinyual <[email protected]>

xinyual added 11 commits September 26, 2025 13:57

add demos

759aab4

Signed-off-by: xinyual <[email protected]>

add missing column

00a1563

Signed-off-by: xinyual <[email protected]>

add appendpipe poc

ddf26a4

Signed-off-by: xinyual <[email protected]>

merge from main

7233111

Signed-off-by: xinyual <[email protected]>

slighty change syntax

a8dd2ca

Signed-off-by: xinyual <[email protected]>

add unresolved plan

c3549f5

Signed-off-by: xinyual <[email protected]>

add IT

9dfc626

Signed-off-by: xinyual <[email protected]>

Merge remote-tracking branch 'origin/main' into addappendPipe

a3e78a8

add tests

f88730f

Signed-off-by: xinyual <[email protected]>

remove useless ut

4c281cc

Signed-off-by: xinyual <[email protected]>

merge from main

70e368d

Signed-off-by: xinyual <[email protected]>

xinyual changed the title ~~Addappend pipe~~ [WIP]: support appendpipecommand in PPL Oct 20, 2025

xinyual added 11 commits October 20, 2025 15:59

fix conflict

93eb5b3

Signed-off-by: xinyual <[email protected]>

remove useless code

9148512

Signed-off-by: xinyual <[email protected]>

remove useless code

345cdeb

Signed-off-by: xinyual <[email protected]>

remove useless code

3d4d865

Signed-off-by: xinyual <[email protected]>

apply spotless

00aba11

Signed-off-by: xinyual <[email protected]>

remove useless chaneg

4c2b311

Signed-off-by: xinyual <[email protected]>

add explain IT

8744cb2

Signed-off-by: xinyual <[email protected]>

fix IT

c48fa34

Signed-off-by: xinyual <[email protected]>

apply spotless

3a23979

Signed-off-by: xinyual <[email protected]>

add doc

f934efa

Signed-off-by: xinyual <[email protected]>

optimize doc

d71e145

Signed-off-by: xinyual <[email protected]>

xinyual marked this pull request as ready for review October 23, 2025 05:28

xinyual requested review from YANG-DB, dai-chen, derek-ho, joshuali925, kavithacm and ps48 as code owners October 23, 2025 05:28

xinyual requested review from LantaoJin, RyanL1997, noCharger, qianheng-aws, ykmr1224 and yuancu as code owners October 23, 2025 05:28

yuancu added the enhancement New feature or request label Oct 31, 2025

yuancu changed the title ~~[WIP]: support appendpipecommand in PPL~~ Support appendpipecommand in PPL Oct 31, 2025

yuancu reviewed Oct 31, 2025

View reviewed changes

ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java Show resolved Hide resolved

Swiddis previously approved these changes Oct 31, 2025

View reviewed changes

add UT

7664cd5

Signed-off-by: xinyual <[email protected]>

xinyual dismissed Swiddis’s stale review via 7664cd5 November 3, 2025 02:19

yuancu reviewed Nov 3, 2025

View reviewed changes

integ-test/src/test/java/org/opensearch/sql/calcite/remote/CalcitePPLAppendPipeCommandIT.java Show resolved Hide resolved

xinyual added 2 commits November 4, 2025 10:20

Merge remote-tracking branch 'origin/main' into addAppendpipe

1624912

fix IT due to performance change

fa2ec9e

Signed-off-by: xinyual <[email protected]>

yuancu previously approved these changes Nov 5, 2025

View reviewed changes

qianheng-aws reviewed Nov 7, 2025

View reviewed changes

xinyual added 2 commits November 12, 2025 10:19

add multiply children check

fe1b23f

Signed-off-by: xinyual <[email protected]>

merge from main

f1b3a81

Signed-off-by: xinyual <[email protected]>

xinyual dismissed yuancu’s stale review via f1b3a81 November 12, 2025 02:22

qianheng-aws approved these changes Nov 12, 2025

View reviewed changes

yuancu approved these changes Nov 12, 2025

View reviewed changes

yuancu merged commit 3a3c8c8 into opensearch-project:main Nov 12, 2025
55 of 56 checks passed

yuancu added the backport 2.19-dev label Nov 12, 2025

opensearch-trigger-bot bot mentioned this pull request Nov 12, 2025

[Backport 2.19-dev] Support appendpipecommand in PPL #4785

Closed

xinyual mentioned this pull request Nov 12, 2025

[Backport 2.19-dev] Support appendpipe command in PPL #4787

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support `appendpipe`command in PPL #4602

Support `appendpipe`command in PPL #4602

Uh oh!

xinyual commented Oct 20, 2025 •

edited

Loading

Uh oh!

yuancu left a comment

Uh oh!

Uh oh!

Uh oh!

yuancu left a comment

Uh oh!

xinyual commented Nov 5, 2025

Uh oh!

qianheng-aws Nov 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support appendpipecommand in PPL #4602

Support appendpipecommand in PPL #4602

Uh oh!

Conversation

xinyual commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

yuancu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yuancu left a comment

Choose a reason for hiding this comment

Uh oh!

xinyual commented Nov 5, 2025

Uh oh!

qianheng-aws Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support `appendpipe`command in PPL #4602

Support `appendpipe`command in PPL #4602

xinyual commented Oct 20, 2025 •

edited

Loading