Support UNION RECURSIVE in OpenSearch PPL

[penghuo]

Description

[Describe what this change achieves]

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Summary by CodeRabbit

Release Notes

New Features

• Added UNION RECURSIVE command support for PPL, enabling recursive queries to traverse hierarchical and graph data
• Supports configurable max_depth and max_rows parameters to control recursion limits
• Enforces schema validation between anchor and recursive blocks
• Requires Calcite query engine

Documentation

• Added user-facing documentation for UNION RECURSIVE command with examples and usage guidelines

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Implements UNION RECURSIVE support in OpenSearch PPL, adding grammar, AST nodes, visitors, Calcite translation with schema validation, recursive relation scoping, and comprehensive tests spanning core, PPL, and integration layers.

Changes

Cohort / File(s)	Summary
Core AST Node & Visitor Infrastructure core/src/main/java/org/opensearch/sql/ast/tree/UnionRecursive.java, core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java, core/src/main/java/org/opensearch/sql/ast/EmptySourcePropagateVisitor.java, core/src/main/java/org/opensearch/sql/analysis/Analyzer.java, core/src/main/java/org/opensearch/sql/ast/dsl/AstDSL.java	New UnionRecursive AST node class with immutable core properties (relationName, maxDepth, maxRows, recursiveSubsearch) and mutable child. Extended visitor infrastructure to support traversal and propagation of UnionRecursive nodes. Added DSL helper for construction. Analyzer throws Calcite-only exception.
Calcite Planning & Recursive Scoping core/src/main/java/org/opensearch/sql/calcite/CalcitePlanContext.java, core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java	Introduced recursive relation stack in CalcitePlanContext with push/pop/lookup operations. CalciteRelNodeVisitor adds visitUnionRecursive handler constructing RepeatUnion with recursive scope management, schema validation (field count, names, type compatibility), and optional max depth/rows limits via LogicalSystemLimit.
Calcite SQL Generation ppl/src/main/java/org/apache/calcite/rel/rel2sql/OpenSearchRelToSqlConverter.java	New OpenSearch-specific RelToSqlConverter extension preserving top-level WITH RECURSIVE clauses. Handles RepeatUnion conversion to SQL CTE syntax and TableSpool delegation.
PPL Grammar & Lexer ppl/src/main/antlr/OpenSearchPPLLexer.g4, ppl/src/main/antlr/OpenSearchPPLParser.g4	Added UNION and RECURSIVE lexer tokens. Introduced unionRecursiveCommand grammar rule with recursiveSubPipeline, unionRecursiveNameArg, and unionRecursiveOption sub-rules. Extended commandName and searchableKeyWord vocabularies.
PPL Parser & Validation ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java, ppl/src/main/java/org/opensearch/sql/ppl/utils/UnionRecursiveValidator.java	AstBuilder.visitUnionRecursiveCommand parses name, max_depth, max_rows options and applies validation. UnionRecursiveValidator ensures recursive relation is referenced, prevents name shadowing/aliasing conflicts, and disallows mixing with other sources.
PPL Utilities ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java	Added UnionRecursive anonymization visitor masking relation name and optionally including max_depth/max_rows.
Unit Tests ppl/src/test/java/org/opensearch/sql/ppl/parser/AstBuilderTest.java, ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java, ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java, ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLUnionRecursiveTest.java	Added tests for AST building, anonymization, syntax parsing (valid/invalid forms), Calcite planning, schema mismatch detection, and SQL generation with CTEs.
Integration Tests integ-test/src/yamlRestTest/resources/rest-api-spec/test/issues/5000.yml	YAML REST test enabling Calcite, creating BOM index with bulk data, executing union recursive query with max_depth=1, validating schema and expected result rows.
Documentation docs/dev/union_recursive_*.md, docs/user/ppl/cmd/union_recursive.md, docs/user/ppl/index.md	Added task-by-task context documentation (Tasks 1–7) detailing implementation phases, user-facing command documentation with syntax/semantics/examples/limitations, and command index entry.

Sequence Diagram(s)

sequenceDiagram
    participant Parser as PPL Parser
    participant Validator as UnionRecursiveValidator
    participant Analyzer as Analyzer
    participant Calcite as CalciteRelNodeVisitor
    participant Context as CalcitePlanContext

    Parser->>Parser: Parse UNION RECURSIVE<br/>create UnionRecursive AST
    Parser->>Validator: validate(recursiveSubsearch,<br/>relationName)
    Validator->>Validator: Traverse plan tree<br/>check recursive reference
    alt Validation fails
        Validator->>Parser: throw SemanticCheckException
    else Validation succeeds
        Validator-->>Parser: validation complete
    end
    
    Parser->>Analyzer: analyze(UnionRecursive)
    alt Non-Calcite engine
        Analyzer->>Analyzer: throw getOnlyForCalciteException
    else Calcite engine
        Analyzer-->>Calcite: proceed to translation
    end
    
    Calcite->>Context: pushRecursiveRelation(name,<br/>rowType)
    Context-->>Calcite: recursive scope established
    
    Calcite->>Calcite: Visit anchor block<br/>construct RelNode
    Calcite->>Calcite: Visit recursive block<br/>rewrite recursive relation<br/>to transientScan
    
    Calcite->>Calcite: validateUnionRecursiveSchema<br/>(anchorRowType,<br/>recursiveRowType)
    alt Schema mismatch
        Calcite->>Calcite: throw SemanticCheckException
    else Schema matches
        Calcite->>Calcite: construct RepeatUnion
    end
    
    alt max_rows specified
        Calcite->>Calcite: wrap with LogicalSystemLimit
    end
    
    Calcite->>Context: popRecursiveRelation()
    Context-->>Calcite: scope closed

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• #4929: Modifies CalciteRelNodeVisitor.java for join handling; this PR adds visitUnionRecursive in the same class, requiring attention to interaction points.
• #4979: Updates CalciteRelNodeVisitor with nested aggregation/hint logic; overlaps with the visitUnionRecursive changes in the same visitor class.

Suggested labels

enhancement, PPL

Suggested reviewers

• anirudha
• ps48
• ykmr1224
• yuancu
• kavithacm
• derek-ho
• joshuali925
• GumpacG

✨ Finishing touches

• 📝 Generate docstrings

---

📜 Recent review details

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 46302b7 and 4ddf392.

📒 Files selected for processing (27)

• core/src/main/java/org/opensearch/sql/analysis/Analyzer.java
• core/src/main/java/org/opensearch/sql/ast/AbstractNodeVisitor.java
• core/src/main/java/org/opensearch/sql/ast/EmptySourcePropagateVisitor.java
• core/src/main/java/org/opensearch/sql/ast/dsl/AstDSL.java
• core/src/main/java/org/opensearch/sql/ast/tree/UnionRecursive.java
• core/src/main/java/org/opensearch/sql/calcite/CalcitePlanContext.java
• core/src/main/java/org/opensearch/sql/calcite/CalciteRelNodeVisitor.java
• docs/dev/union_recursive_task1_context.md
• docs/dev/union_recursive_task2_context.md
• docs/dev/union_recursive_task3_context.md
• docs/dev/union_recursive_task4_context.md
• docs/dev/union_recursive_task6_context.md
• docs/dev/union_recursive_task7_context.md
• docs/dev/union_recursive_tasks.md
• docs/user/ppl/cmd/union_recursive.md
• docs/user/ppl/index.md
• integ-test/src/yamlRestTest/resources/rest-api-spec/test/issues/5000.yml
• ppl/src/main/antlr/OpenSearchPPLLexer.g4
• ppl/src/main/antlr/OpenSearchPPLParser.g4
• ppl/src/main/java/org/apache/calcite/rel/rel2sql/OpenSearchRelToSqlConverter.java
• ppl/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizer.java
• ppl/src/main/java/org/opensearch/sql/ppl/utils/UnionRecursiveValidator.java
• ppl/src/test/java/org/opensearch/sql/ppl/antlr/PPLSyntaxParserTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/calcite/CalcitePPLUnionRecursiveTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/parser/AstBuilderTest.java
• ppl/src/test/java/org/opensearch/sql/ppl/utils/PPLQueryDataAnonymizerTest.java

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}